Big data platform risk knowledge graph construction method

Abstract

The invention provides a big data platform risk knowledge graph construction method. The method comprises the steps of constructing a big data platform risk knowledge base; the method comprises the following steps: respectively constructing an ontology of a risk map from bottom to top, extracting a relationship between entities of the big data platform risk knowledge map according to the definition of the ontology in the big data platform risk knowledge map, pairing the entities with risk attributes to form a risk knowledge triple, extracting features of each entity, and establishing a risk knowledge triple; the method comprises the following steps of: obtaining features of entities, calculating similarity between the entities by utilizing the features, determining matched entities by utilizing the similarity, carrying out fusion processing, reforming risk knowledge triple storage, automatically constructing a big data platform risk knowledge graph, carrying out knowledge completion, and carrying out quality evaluation on the big data platform risk knowledge graph subjected to knowledge completion; and dynamically updating the knowledge complemented big data platform risk knowledge graph to correct and supplement missing information. The big data platform risk knowledge graph constructed by the method has universality and expandability.

Description

Technical Field

[0001] The present application belongs to the technical field of big data risk assessment, and specifically relates to a method for constructing a risk knowledge graph for a big data platform. Background Art

[0002] In recent years, in order to optimize the economic structure and cope with the downward pressure on the economy, countries around the world, including my country, have begun to support the development of the "digital economy" and regard it as a "new engine" for high-quality economic and social progress. Big data is the cornerstone of the development of the "digital economy" and has unpredictable and huge value.

[0003] Massive big data needs to rely on big data platforms to realize their own value. Big data platforms have also been widely used in finance, medical care, retail, communications and other fields due to their powerful data storage and processing capabilities. These platforms provide enterprises with in-depth insights and decision support by integrating data from social media, internal enterprise transaction processing systems, IoT devices and public data sources.

[0004] However, because big data platforms process huge amounts of data and a wide variety of data, they have become the main target of hackers and internal threats, and face numerous security risks. These risks include, but are not limited to, data leakage, in which sensitive information may be obtained by unauthorized third parties; illegal access, in which malicious users bypass security measures through technical means to access, modify or delete important data; data corruption, in which data is damaged during storage or transmission due to software defects, hardware failures or human errors; system failures, in which the entire big data platform cannot be used normally in the short term due to software errors, hardware failures or network problems. These security risks not only threaten the integrity and confidentiality of data, but may also affect the operational efficiency and reputation of the enterprise.

[0005] On the other hand, traditional risk management methods often rely on expert experience and qualitative analysis, so it is difficult to cope with the characteristics of large data volume, fast update and multiple types in the big data environment. This method has limitations in risk identification and quantitative assessment, and it is difficult to fully and accurately grasp the security risk status of the big data platform, which affects the effectiveness and timeliness of risk management. Specifically, traditional methods have significant limitations in the following aspects: 1) Insufficient ability to process large-scale data: When the amount of data increases to TB or even PB level, the performance of traditional tools in storing, processing and analyzing these data is limited, and it cannot effectively support risk identification and assessment work; 2) Unable to cope with the challenge of data diversity: In the big data environment, there are many types of data, including but not limited to text, images, log files, etc. Traditional methods are mainly aimed at structured data, and have limited processing capabilities for semi-structured and unstructured data, making it difficult to extract valuable risk information from them; 3) Lack of real-time and dynamic characteristics: The risk status in the big data environment is dynamically changing and requires real-time monitoring and analysis. However, traditional risk management methods often rely on periodic assessments and lack real-time analysis and response capabilities, resulting in the inability to timely discover and respond to emerging risks; 4) Limited quantitative analysis capabilities: Traditional methods rely on expert experience and qualitative analysis, and often find it difficult to accurately quantify risks. In a big data environment, risk factors are complex and changeable, and it is difficult to accurately assess the severity and impact of risks based on qualitative analysis alone, which affects the scientific nature and effectiveness of risk management decisions. These limitations not only affect the comprehensiveness and accuracy of risk management, but may also prevent companies from timely discovering and responding to potential security threats, thereby increasing operational risks and affecting the stability and sustainable development of the company. Therefore, in the face of new challenges in the big data environment, new risk management methods and technologies are urgently needed to improve the coverage, accuracy and real-time nature of risk identification, and ensure that companies can effectively respond to the increasingly complex security risk environment.

[0006] Knowledge graph was first proposed by Google in 2012. Its essence is a large-scale semantic network. By building various types of relationships between entities, it can not only effectively organize and utilize a large amount of complex information, but also provide in-depth insights and promote the discovery and application of knowledge. In the risk management of big data platforms, the application of knowledge graph is particularly important. It can integrate risk information scattered in different data sources and build an interconnected and dynamic information network, thereby helping enterprises to identify and evaluate potential risks more systematically.

[0007] To build a knowledge graph, we first need to extract key entities, such as risk points, influencing factors, control measures, etc., as well as the relationships between these entities, from a large amount of data. This process usually involves natural language processing, data mining, and machine learning techniques to automatically identify valuable information in the data and structure it. In this way, the knowledge graph can visualize complex risk information, allowing risk managers to intuitively understand the interactions and influence mechanisms between risk factors.

[0008] Applying knowledge graphs to risk management on big data platforms can significantly improve the accuracy and efficiency of risk identification and assessment. Traditional risk management methods often rely on the subjective judgment of experts, while knowledge graphs can reveal undiscovered connections between risk factors through data-driven methods, supporting more accurate risk prediction. In addition, knowledge graphs can also automate risk management and help companies quickly respond to emerging threats by updating risk information in real time.

[0009] Knowledge graphs can not only help companies identify current risks, but also predict possible future risks based on historical data and risk models, providing a scientific basis for risk prevention and control and decision-making. By analyzing the relationship between risk factors, companies can allocate resources more effectively, optimize risk control measures, and achieve proactive risk management.

[0010] In the fields of finance, cybersecurity, supply chain management, etc., the application of knowledge graphs has demonstrated its great potential in risk management. For example, in the field of cybersecurity, knowledge graphs can integrate security event information from different data sources, reveal the relationship between attack patterns and vulnerability exploits, and support security teams to conduct effective threat hunting and emergency response. In supply chain management, knowledge graphs can help companies identify potential risk points in the supply chain, such as supplier risks, logistics risks, etc., and support more accurate and timely risk response strategies.

[0011] Big data is the cornerstone of the development of the "digital industry". It relies on big data platforms for storage and processing to exert its own value. However, at present, big data platforms face serious security risks. Due to the complex architecture of big data platforms, diverse asset types, and numerous potential threats, comprehensive measurement of the security risks of big data platforms requires in-depth search of causal dependencies between multiple entities and events inside and outside the platform, as well as comprehensive correlation analysis of security data from multiple aspects. In addition, with the continuous escalation of advanced persistent threats, a large number of network attacks are often hidden in complex relational network data and are difficult to be effectively identified. How to fully grasp the security risk status of big data platforms and accurately identify and locate risks is a security supervision and governance problem that all big data platforms at home and abroad need to solve urgently.

[0012] The existing technology records the knowledge graph construction scheme. The construction of vertical knowledge of network security must consider the depth of knowledge and the overall hierarchy. Therefore, a top-down approach is adopted: first, a network security ontology is constructed. Based on the constructed ontology, network security information is extracted from structured and unstructured data.

[0013] 1. Construction of network security ontology

[0014] like Figure 1 As shown, the network security ontology is assets, vulnerabilities and attacks, which contain five entity types in total.

[0015] (1) Vulnerability: Each record in the vulnerability database corresponds to an instance of a vulnerability type. Each vulnerability has its own unique CVE ID.

[0016] (2) Assets: Assets include software and operating system (OS).

[0017] (3) Software: This is a subclass of assets (such as Adobe Reader).

[0018] (4) Operating System: This is a subclass of Asset (such as Ubuntu 14.04)

[0019] (5) Attack: Most attacks can be viewed as intrusions targeting a vulnerability, e.g., the attack process may be a process of exploiting a vulnerability.

[0020] (II) Data Source

[0021] Vulnerabilities include existing vulnerabilities that can be collected, and attacks include the most popular attack techniques. Among them, the sources of vulnerabilities include CVE, NVD, SecurityFocus, CXSECURITY, Secunia, CNVD, CNNVD, and SCAP. The data sources of attacks mainly include two categories: one is from information security websites, including Pediy BBS, Freebuf, Kafan BBS, and Open Web Application Security Project (OWASP); the other is from the information response centers built by enterprises, including 360 Security Response Center (360SRC) and Alibaba Security Response Center (ASRC).

[0022] 3. Extraction of network security related entities: machine learning based methods

[0023] Conditional Random Field (CRF) is an undirected graph model based on statistical sequence recognition and segmentation. The main idea of ​​this model comes from the maximum entropy model, and its simplest form is linear CRF, where the nodes in the model form a linear structure, and linear CRF corresponds to a finite state machine, which is very suitable for labeling linear data sequences.

[0024] The named entity recognition problem can be defined as a sequence labeling problem, that is, whether the observed word belongs to a predefined set of features. Conditional random fields are probabilistic models for sequence labeling. It has no independent assumptions, can select features arbitrarily, and globally normalizes all features to obtain a global optimal solution. It retains the advantages of the conditional probability framework, such as the maximum entropy Markov model. It also solves the problem of labeling bias. Therefore, the conditional random field model is suitable for named entity recognition.

[0025] Stanford NER provides a general implementation of the linear chain conditional random field (CRF) sequence model, so it is also called a CRF classifier. Use Stanford NER to extract entities related to network security. Stanford NER provides many options for features, and uses Stanford NER base to train the extraction model.

[0026] It is very important to select features when building a model. Here, a feature should be selected that can better identify cybersecurity-related entities. A good combination of features is the key to training a good extraction model. Stanford NER provides more than 70 features that can be used to train a model. Determining suitable features is not an easy task because Stanford does not have much documentation to explain these features. Existing feature selection algorithms will not be of much help. You must analyze the existing features yourself, select the features that you think are useful for training the model, and then verify them through experiments. After many experiments, a feature set for training the name recognizer was finally determined. Training NER based on this feature set achieved good recognition results. The feature set used to train the name recognizer is as follows.

[0027] (1) UseNGrams: Use n-grams to extract features, i.e., substrings of words.

[0028] (2) MaxNGramLeng: The value type of this feature is integer. If the value of this feature is positive, n-grams greater than this value will not be used in the model. Here, the value of MaxNGramLeng is set to 6.

[0029] (3) UsePrev: This can provide<previous word,class of previous word> , and is enabled with other options such as<previous tag,class> This results in a pair of<previousword,class of previous word> This feature is very useful when there are consecutive words belonging to the same class.

[0030] (4) UseNext: Very similar to the UsePrev feature.

[0031] (5) UseWordPairs: This feature is based on two word pairs -<Previous word,current word,class> and<current word,next word,class> .

[0032] (6) UseTaggySequences: This is an important feature that uses the sequence of classes instead of the set of words, i.e., using the first, second and third order classes and tag sequences as interaction features.

[0033] (7) UseGazettes: If true, the file is identified as an entity dictionary by the next feature named "gazette".

[0034] (8) Gazette: This value can be one or more file names (separated by commas, semicolons, or spaces). If loading a public entity dictionary from these files, each line should be an entity class name, followed by a space, followed by an entity.

[0035] (9) CleanGazette: If this value is true, this feature will trigger only when the entire word is matched in the dictionary. If there is a word "Windows 7" in the dictionary, then the entire word should be matched in the dictionary.

[0036] (10) SloppyGazette: If this value is true, a partial match between a word and a word in the dictionary can also trigger this feature, such as "Windows" can match "windows 7".

[0037] Dictionary-related features are used here, and Stanford NER provides two specific implementations of dictionary-related features. Experiments have shown that the gazette and cleanGazette options are very good choices because they improve the recognition accuracy of software and operating system classes.

[0038] In order to use this feature, it is necessary to aggregate relevant information from the influence platform field in the vulnerability database and build an entity dictionary. The first column in the dictionary is the entity type, and the second column corresponds to the specific entity.

[0039] The above existing methods based on risk assessment framework or security vulnerability scanning often focus on static security assessment, lack an in-depth understanding of the security risk context, and cannot support in-depth search and correlation analysis of dependencies between multiple entities and events inside and outside the platform, as well as accurate risk identification and positioning. At the same time, the existing methods are not universal and scalable for different big data platforms, and the coverage of risk identification is low, making it difficult to identify and respond to complex advanced persistent threats and internal threats. In addition, the existing methods lack the ability to analyze and respond to risks in real time, and rely on expert experience and qualitative analysis, making it difficult to accurately quantify risks. Furthermore, the existing methods are also difficult to identify and respond to complex advanced persistent threats and internal threats. Summary of the invention

[0040] In order to solve the above problems existing in the prior art, this application provides a method for constructing a risk knowledge graph on a big data platform. The technical problem to be solved by this application is achieved through the following technical solutions:

[0041] A method for constructing a risk knowledge graph for a big data platform includes:

[0042] S100, collecting relevant risk knowledge of the big data platform and organizing and cleaning the data to build a big data platform risk knowledge base; the big data platform risk knowledge base includes structured risk knowledge and unstructured risk knowledge;

[0043] S200, adopts the method of combining manual definition with domain expert knowledge to build the ontology of asset, data and business risk graphs from bottom to top, and obtains the ontology of risk knowledge graph of big data platform;

[0044] S300, based on the definition of the ontology in the risk knowledge graph of the big data platform, and using the knowledge extraction model to extract the entities and the relationships between the entities in the risk knowledge graph of the big data platform, and pair the entities with the risk attributes based on the risk knowledge base of the big data platform to form risk knowledge triples; the risk knowledge triples include entities, relationships between entities and entity attributes;

[0045] S400, extracting the features of each entity in the risk knowledge triple, and using the features to calculate the similarity between the entities, using the similarity to determine matching entities, and fusing the matching entities to re-form the risk knowledge triple;

[0046] S500 stores the reorganized risk knowledge triples and automatically constructs them into a risk knowledge graph on the big data platform;

[0047] S600, completing the knowledge of the big data platform risk knowledge graph, and performing quality assessment on the knowledge-completed big data platform risk knowledge graph;

[0048] S700 dynamically updates the risk knowledge graph of the knowledge completion big data platform to correct and supplement missing information.

[0049] Beneficial effects:

[0050] (1) This application addresses the current research gap in the academic and industrial circles regarding risk knowledge graphs for big data platforms. It systematically proposes a technical solution for the full-process construction of risk knowledge graphs for big data platforms, from ontology construction to knowledge extraction, knowledge fusion, and finally to knowledge storage. At the same time, it adds a knowledge completion and dynamic update mechanism, and establishes a big data platform and data risk knowledge graph library including a big data platform asset risk graph, a data risk graph, and a business risk graph. This strongly supports the security risk identification and quantitative assessment of big data assets, the entire life cycle of data, and potential attack behaviors, and provides a practical idea for the application of knowledge graphs in the security field.

[0051] (2) This application constructs a more fine-grained cybersecurity risk ontology model for the three major cybersecurity sub-scenarios of big data platform assets, data, and business, which can more clearly and specifically describe the various risk entities and their relationships within the scenario, helping managers to fully grasp the security risk status of the big data platform and accurately identify and locate risks.

[0052] (3) Aiming at the situation where traditional machine learning models usually require a lot of feature engineering in named entity recognition tasks and relationship extraction tasks, this application proposes a deep learning method that combines the BERT-BiLSTM-CRF model and the PERT-FC model. The introduction of a pre-trained language model and the integration of an open source knowledge graph can significantly reduce the need for complex feature engineering and provide more accurate entity recognition and relationship extraction in complex texts.

[0053] (4) In view of the fact that traditional risk management methods often rely on periodic assessments and are unable to promptly detect and respond to emerging risks, this application proposes a knowledge graph dynamic update solution based on a stream processing engine deployment. The stream processing technology is used to map the real-time changes in the knowledge base to the corresponding parts of the risk graph to reflect the latest platform security risk status.

[0054] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart of a method for constructing a risk knowledge graph for a big data platform provided in an embodiment of the present application;

[0056] Figure 2 This is a flow chart of the big data platform risk knowledge base construction phase provided by the embodiment of the present application;

[0057] Figure 3 It is a flow chart of the risk knowledge graph ontology construction phase of the big data platform provided in the embodiment of the present application;

[0058] Figure 4 It is the asset risk map provided by the embodiment of the present application;

[0059] Figure 5 It is a data collection risk map provided by an embodiment of the present application;

[0060] Figure 6 It is a data storage risk map provided by an embodiment of the present application;

[0061] Figure 7 It is a data transmission risk map provided by an embodiment of the present application;

[0062] Figure 8 It is a data processing risk map provided by an embodiment of the present application;

[0063] Fig. 9 It is a data sharing risk map provided by an embodiment of the present application;

[0064] Fig.10 It is a data backup, recovery and deletion risk map provided by the embodiment of the present application;

[0065] Fig.11 It is a business risk map provided by an embodiment of the present application;

[0066] Fig.12 This is a flow chart of the knowledge extraction phase of the risk knowledge graph of the big data platform provided in the embodiment of the present application;

[0067] Fig.13 It is a data collection risk map risk knowledge manual annotation map provided in the embodiment of the present application;

[0068] Fig.14 It is a BIO ternary annotation example diagram provided in the embodiment of the present application;

[0069] Fig.15 This is the overall architecture diagram of the BERT-BiLSTM-CRF model provided in the embodiment of the present application;

[0070] Fig.16 It is a flow chart of the knowledge fusion phase of the risk knowledge graph of the big data platform provided in the embodiment of the present application;

[0071] Fig.17 It is a flow chart of the knowledge storage phase of the risk knowledge graph of the big data platform provided in the embodiment of the present application;

[0072] Fig.18 This is a simplified example diagram of an asset risk map provided in an embodiment of the present application;

[0073] Fig.19 It is a Cypher language complex association query example diagram provided in an embodiment of the present application;

[0074] Fig. 20 This is a flow chart of the knowledge completion phase of the risk knowledge graph of the big data platform provided in the embodiment of the present application;

[0075] Fig.21 It is a flow chart of the dynamic update phase of the risk knowledge graph of the big data platform provided in the embodiment of the present application. DETAILED DESCRIPTION

[0076] The present application is further described in detail below in conjunction with specific embodiments, but the implementation methods of the present application are not limited thereto.

[0077] This application designs a method for constructing a risk knowledge graph for a big data platform to help managers fully grasp the security risk status of the big data platform and support the risk identification and quantitative assessment of the big data platform. According to the application scope of the knowledge graph, the risk knowledge graph of the big data platform belongs to the "domain knowledge graph", so this application adopts a "top-down" graph construction method for construction. First, this application perceives the assets, data, and business of the big data platform, and obtains the risk knowledge base of the big data platform after data cleaning; secondly, this application adopts the method of artificial definition combined with the knowledge of experts in the big data field to construct the ontology layer of the three major risk maps of the big data platform assets, data, and business, ensuring the professionalism and accuracy of the risk ontology, relationship, and attribute, and ensuring the quality of ontology construction; then, the deep learning method combined with the pre-trained language model and the rule matching method are used to extract entities, relationships, and attributes from the big data platform risk knowledge base based on the constructed ontology model to form risk knowledge triples; then, this application adopts the entity resolution method to fuse the risk knowledge triples, eliminate the noise data and redundant data in the massive risk knowledge, and further improve the knowledge quality; finally, this application uses the Neo4j graph database for knowledge storage, which can quickly respond to complex association relationship queries and support all-round, multi-dimensional graph visualization, which helps to conveniently and quickly conduct complex risk association analysis and comprehensively and intuitively grasp the security risk status of the big data platform. In addition, this application completes the risk knowledge graph of the big data platform based on the hyperbolic graph attention network and contrastive learning method, thereby mining various hidden associations that may exist in the graph, making the graph more accurate and complete. At the same time, this application also introduces a dynamic update mechanism for the knowledge graph of the associated big data platform, so that the graph can reflect the latest status of the big data platform in a timely manner. The risk knowledge graph of the big data platform constructed by this application is universal and suitable for most big data platforms built on the Kubernetes architecture, and the risk ontology of this application is scalable, and the big data platform managers can adjust it according to actual conditions, which is highly flexible.

[0078] The meanings of platform, assets, data and business in this application are:

[0079] Platform: Big data platform; Assets: Asset objects in the platform; Data: Data assets in the platform; Business: Business in the platform;

[0080] The knowledge graph can be formally defined as where ε represents the set of entities, represents a collection of relations, Represents a set of edges. Each triple (x, r, y) represents an edge in the knowledge graph, indicating that a head entity x and a tail entity y are connected by a relationship r.

[0081] refer to Figure 1 As shown, the present application provides a method for constructing a risk knowledge graph of a big data platform, including:

[0082] S100, collecting relevant risk knowledge of the big data platform and organizing and cleaning the data to build a big data platform risk knowledge base; the big data platform risk knowledge base includes structured risk knowledge and unstructured risk knowledge;

[0083] Before formally constructing the risk knowledge graph of the big data platform, it is first necessary to collect and organize the relevant risk knowledge of the big data platform to form a risk knowledge base of the big data platform.

[0084] S200, adopts the method of combining manual definition with domain expert knowledge to build the ontology of asset, data and business risk graphs from bottom to top, and obtains the ontology of risk knowledge graph of big data platform;

[0085] Since there is currently no research on risk ontology of big data platforms available for reference and reuse at home and abroad, this application adopts the method of "manual definition combined with domain expert knowledge" to complete the construction of the ontology layer of the three major risk knowledge graphs of assets, data, and business from the bottom up.

[0086] S300, extracting entities and relationships between entities in the risk knowledge graph of the big data platform using the knowledge extraction model, and pairing entities with risk attributes based on the risk knowledge base of the big data platform to form risk knowledge triples; the risk knowledge triples contain entities, relationships between entities, and entity attributes;

[0087] S400, extracting the features of each entity in the risk knowledge triple, and using the features to calculate the similarity between the entities, using the similarity to determine matching entities, and fusing the matching entities to re-form the risk knowledge triple;

[0088] S500 stores the reorganized risk knowledge triples and automatically constructs them into a risk knowledge graph on the big data platform;

[0089] S600, completing the knowledge of the big data platform risk knowledge graph, and performing quality assessment on the knowledge-completed big data platform risk knowledge graph;

[0090] S700 dynamically updates the risk knowledge graph of the knowledge completion big data platform to correct and supplement missing information.

[0091] As an optional implementation of the present application, refer to Figure 2 , S100 includes:

[0092] S110, collecting data information and all possible sensing means for sensing the data information from the big data platform, wherein the data information includes: assets, data, and business;

[0093] S120, using a penetration testing tool set that aggregates all possible perception means to automatically and actively perceive the data information, and combining expert knowledge to supplement the risk knowledge that cannot be perceived in the data information to obtain risk knowledge; the risk knowledge includes: structured risk knowledge and unstructured risk knowledge;

[0094] In this step, penetration testing tools such as Nmap, Wireshark, and Metasploit can be deployed in the form of probes. Of course, any tool with the same penetration testing function can also be used to regularly and automatically sense the assets, data, and services within the platform and their vulnerabilities. At the same time, the domain expert knowledge of the big data platform is combined to supplement the risk knowledge that cannot be obtained through perception, such as the relationship between assets, data, and services and risk attributes.

[0095] S130, performing data cleaning operations on the risk knowledge and forming a risk knowledge base of the big data platform.

[0096] The data cleaning in this step includes deduplication, error removal and other data cleaning methods.

[0097] As an optional implementation of the present application, refer to Figure 3 , S200 includes:

[0098] S210, constructing the ontology of the asset risk map from the bottom up by combining manual definition with domain expert knowledge, wherein the asset risk map uses the asset as an ontology and defines the association relationship and risk attributes between the ontologies;

[0099] In the asset risk map, this application uses common asset objects in the platform as the ontology and defines the relationship between them, such as Figure 4 As shown. The six categories of entities in the asset risk map include servers, containers, data, interfaces, network devices, and security devices. Among them, servers run containers, which store and manage, authenticate and transmit, process and analyze data. Containers also provide internal and external interfaces. Network devices support servers, and security devices protect servers and network devices. The risk attribute definitions of each entity are not detailed here. For details, please refer to Figure 4 shown.

[0100] S220, using manual definition combined with domain expert knowledge to construct the ontology of the data risk map from the bottom up, the data risk map is divided into six risk sub-graphs of data collection, data storage, data transmission, data processing, data sharing, data backup, recovery and deletion according to the full life cycle of the data on the big data platform, and the ontology and risk attributes of each risk sub-graph are defined;

[0101] In the data risk map, this application divides it into six risk sub-maps based on the full life cycle of big data platform data: data collection, data storage, data transmission, data processing, data sharing, data backup, recovery and deletion. Their ontology layer definitions are as follows: Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10As shown. The six categories of ontology of the data collection risk map include business department, collection task, preprocessing task, data source, data lake and security policy. Among them, the business department publishes collection tasks and preprocessing tasks, the collection task collects data source, the data source connects data to the data lake, and the preprocessing task processes the data lake. The collection task and the data source adopt certain security policies; the six categories of ontology of the data storage risk map include business department, data warehouse, data quality detection task, data table, data field and security policy. Among them, the business department manages and accesses the data warehouse, and publishes data quality detection tasks to detect the data table stored in the data warehouse. The data table contains data fields. The data warehouse, data table and data field all adopt certain security policies; the five categories of ontology of the data transmission risk map include sender, receiver, transmission link, data table and security policy. Among them, the sender sends data to the transmission link, the transmission link transmits the data table, and the receiver receives data from the transmission link. The sender, transmission link and receiver all adopt certain security policies; the five categories of ontology of the data processing risk map include business department, business system, business, data table and security policy. Among them, the business department manages the business system and creates business, the business system processes the created business, the business is associated with the data table, and the business system adopts certain security policies; the six categories of ontology of the data sharing risk map include the sharer, the receiver, the data table, the data field, the attack method and the security policy. Among them, the sharer and the receiver provide and receive, publish and obtain, exchange and exchange data tables respectively. The data table contains data fields. The attack method can use the data table and data fields to attack. The sharer adopts certain security policies to defend against possible attack methods. The six categories of ontology of the data backup, recovery and deletion risk map include the business department, data deletion task, data recovery task, original data table, backup data table and security policy. Among them, the business department publishes data deletion tasks and data recovery tasks, the backup data table backs up the original data table, the data deletion task deletes the original data table and the backup data table, the data recovery task restores the original data table, and the data deletion task and the data recovery task both adopt certain security policies. The risk attribute definitions of each ontology are not repeated here. For details, please refer to Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 shown.

[0102] This application proposes ontology-level definitions of the three major risk maps of assets, data, and business, in which all descriptions of ontologies, relationships, and attributes involved can be replaced by synonyms.

[0103] S230, using manual definition combined with domain expert knowledge to construct the ontology of the business risk map from bottom to top, wherein the business risk map describes the association between businesses and between businesses and data, and defines various ontologies and risk attributes;

[0104] In the business risk map, this application focuses on describing the relationship between business and business, business and data, such as Fig.11 The three types of ontology in the business risk map include business departments, businesses, and data tables. Among them, business departments carry out different businesses, different businesses collaborate with each other, and businesses are related to data tables. The risk attribute definitions of various ontologies are not described here. For details, please refer to Fig.11 shown.

[0105] S240, performing quality assessment on each ontology in the asset risk map, the data risk map and the business risk map to obtain assessment results of each ontology.

[0106] This application proposes an ontology definition for the risk knowledge graph of a big data platform, which accurately describes the security risk status and correlation relationships of the assets, data, and business of the big data platform. The risk ontology definition is professional, accurate, and highly usable, and has good versatility and scalability, which fills the gap in the current research on risk ontology of big data platforms at home and abroad.

[0107] This step can conduct a detailed evaluation of the quality of the ontology model constructed above from five aspects: completeness, scalability, clarity, compatibility, and consistency.

[0108] As an optional implementation of the present application, refer to Fig.12 , S300 includes:

[0109] S310, using the unstructured risk knowledge in the risk knowledge base of the big data platform as risk corpus, manually annotating the risk corpus with the help of Label Studio open source data annotation tool according to the definition of the ontology in the risk knowledge graph of the big data platform, and annotating the annotated risk corpus based on the BIO ternary annotation rule to obtain the annotated risk corpus;

[0110] Take the data collection risk map as an example. Fig.13 After the manual annotation is completed, this application continues to further annotate the risk knowledge based on the BIO ternary annotation rule. Among them, B represents the starting word of the annotation, I represents the annotated word after the start, and O represents the word that does not belong to the annotation. The processed annotation example is as follows Fig.14 shown.

[0111] This application proposes to use the Label Studio open source data labeling tool for manual labeling, where "LabelStudio" can be replaced by any tool with the same data labeling function.

[0112] S320, inputting the annotated risk corpus into the BERT submodule included in the knowledge extraction model to perform dynamic vector representation to obtain encoded first data; and inputting the first data into the BiLSTM submodule included in the knowledge extraction model to learn long-distance text information, and then using CRF to obtain the global optimal annotation sequence, so as to output the entities contained in the risk corpus;

[0113] refer to Fig.15 The knowledge extraction model is implemented based on the BERT-BiLSTM-CRF model and consists of two parts: named entity recognition and relationship extraction. The named entity recognition part includes three sub-modules: BERT, BiLSTM, and CRF, and the relationship extraction part includes two sub-modules: PERT and FC. The BERT sub-module is obtained by fine-tuning the BERT sub-module by integrating the open source network threat intelligence knowledge graph.

[0114] In the named entity recognition part, this application implements the recognition of entities in unstructured risk knowledge based on the BERT-BiLSTM-CRF model, and integrates the open source network threat intelligence knowledge graph LLM-TIKG to fine-tune the BERT model to adapt to the network security entity extraction task of this application. The open source network threat intelligence knowledge graph LLM-TIKG contains important network threat knowledge extracted from unstructured network threat intelligence reports, which can be used as additional training data to alleviate the data sparsity problem in the field of network security.

[0115] This application adds an additional knowledge encoder layer inside the BERT model to enable it to have the ability to process knowledge information in the knowledge graph. The knowledge encoder converts the input knowledge graph into a high-dimensional vector representation, interacts with the text encoder, and captures the relationship between text and knowledge through the attention mechanism, thereby improving the model performance. In the relationship extraction part, this application implements the extraction of entity relationships in unstructured risk knowledge based on the PERT-FC model. PERT is the Permuted Language Model proposed by the iFlytek Laboratory of Harbin Institute of Technology in 2022, and its model structure is basically the same as BERT. In terms of entity recognition, PERT performs slightly worse than BERT. This is because the proportion of labeled entities in sentences in the field of network security is large, and the arrangement of words may cause changes in the meaning of sentences. For relational data, the proportion of labels in sentences is small, so compared with MLM, the PerLM model is more suitable for relational data. Therefore, this application uses PERT as an encoder and a fully connected layer (FC) as a decoder to extract the relational data of entities.

[0116] This application also proposes to combine the BERT-BiLSTM-CRF model and the PERT-FC model to realize named entity recognition and relationship extraction, and use the PyTorch framework to build the model, and integrate the open source network threat intelligence knowledge graph LLM-TIKG to fine-tune the BERT model. Among them, not only can the "BERT-BiLSTM-CRF model and PERT-FC model" be built with the help of other deep learning frameworks such as TensorFlow, but the "BERT-BiLSTM-CRF model and PERT-FC model" can also be replaced by any mainstream model with the same named entity recognition and relationship extraction functions, and the "open source network threat intelligence knowledge graph LLM-TIKG" is similar. The fusion method of the knowledge graph based on modifying the internal encoding structure and the pre-trained model can also be replaced by any mainstream method with the same fusion function.

[0117] S330, dividing the entity into a subject and an object, and inputting the entity into a PERT submodule together with the risk corpus to extract context features of the relational data to obtain encoded second data, and inputting the second data into an FC submodule to obtain a first risk knowledge triple containing entities and risk relations between entities;

[0118] This application selects precision, recall and the harmonic mean F1 value of the two as evaluation indicators of model performance, and stipulates that named entity recognition is correct only when the entity boundaries recognized by the model are consistent with the true boundaries. If the relationship category and the head and tail entities are correct, the extracted relationship is considered correct.

[0119] S340, based on the location and context information of the entity, the entity in the risk knowledge triplet is matched with the risk attribute in the risk knowledge base of the big data platform by using rule matching to form a second risk knowledge triplet including the entity and its attributes.

[0120] The risk knowledge triplet includes a first risk knowledge triplet and a second risk knowledge triplet. The first risk knowledge triplet is expressed in the form of G xry =(entity x, relation r, entity y), G xry Indicates that there is a relationship r between entity x and entity y. The second risk knowledge triple is expressed as G xpv =(entity x, risk attribute p, attribute value v), G xpv It means that entity x has a certain risk attribute p and its attribute value is v.

[0121] This application proposes a knowledge extraction scheme that combines the BERT-BiLSTM-CRF model and the PERT-FC model. With the additional network threat knowledge provided by the open source network threat intelligence knowledge graph LLM-TIKG and the powerful deep semantic understanding ability of the pre-trained language model, accurate risk entity recognition and relationship extraction are achieved with only a small amount of annotated risk corpus.

[0122] As an optional implementation of the present application, refer to Fig.16 , S400 includes:

[0123] S410, deduplicating the risk knowledge triples, extracting features of entities in the deduplicated risk knowledge triples, and combining related features of the entities to form all features of the entities;

[0124] S420, for all features of the entity, using TF-IDF weighted cosine similarity as a metric, calculating similarities between entities, determining entities with similarities greater than a similarity threshold, and taking them as matched entities;

[0125] This step builds features that can be used for comparison algorithms based on the attributes of entities in the risk knowledge triples and entities associated with them, including features directly extracted from triples, such as entity names, attributes, attribute values, etc.; or features obtained by calculation, such as the edit distance between entity names, attribute names, attribute values, etc.; or other features of entities, such as the ontology category to which the entity belongs, etc. This application can also continuously optimize the similarity threshold through experiments.

[0126] This application proposes a knowledge fusion solution based on feature extraction and cosine similarity calculation. By merging and unifying the entity names and attributes of different risk entities that match each other, it eliminates noise data and redundant data in massive risk knowledge and further improves the quality of knowledge.

[0127] S430, unifying the matched entities into the same entity name, merging the entity attributes, and obtaining a risk knowledge triple after matching processing;

[0128] S440, de-duplicate the risk knowledge triples after matching, and perform quality assessment to obtain an assessment result.

[0129] This application implements a fine-grained fusion strategy for different entities that match each other in the risk knowledge triples, that is, the matching entities are unified into the same entity name, and the attributes of the matching entities are merged, giving priority to retaining the most complete or most credible information. All risk knowledge triples are deduplicated again, and data quality indicators such as accuracy, consistency, and completeness are evaluated and reported.

[0130] As an optional implementation of the present application, refer to Fig.17 , S500 includes:

[0131] S510, storing the reorganized risk knowledge triples based on the Neo4j graph database;

[0132] S520, using the Py2neo library to convert the reconstructed risk knowledge triples into nodes and edges in the risk knowledge graph of the big data platform, so as to automatically construct the risk knowledge graph of the big data platform; wherein the entities in the risk knowledge triples are mapped to nodes in the risk knowledge graph of the big data platform, the risk relationships between the entities are mapped to edges in the risk knowledge graph of the big data platform, and the risk attributes are mapped to attributes of the nodes;

[0133] S530, using a visualization tool Neo4j Bloom, visually display the risk knowledge graph of the big data platform.

[0134] Take a simplified asset risk map as an example. Fig.18 Neo4j Bloom is a data exploration tool that helps big data platform managers visualize data in risk maps and allows managers to easily navigate and query data without any query language or programming. Managers can write patterns similar to natural language questions to retrieve data, and when missing information or bad data is found, Bloom can be used directly to edit, update or correct the map.

[0135] After obtaining the risk knowledge graph of the big data platform, users can query it. In the risk knowledge graph of the big data platform, complex association queries can be implemented based on the Cypher query language to support various complex risk association analysis and decision-making needs, such as Fig.19As shown, the Node node obtains the information of the Container node that is not directly connected to it through a two-hop "run" relationship query. At the same time, indexes are created for commonly used query attributes, especially node attributes that are often used as query conditions, to further improve query efficiency. In addition, this application also uses Neo4j's caching mechanism to improve the access speed of hot data, shorten the response time of repeated queries by reasonably configuring page cache and object cache, and use Neo4j's monitoring tool to monitor the operating status of the database, and promptly discover and solve performance bottlenecks.

[0136] For the Neo4j graph database, this application enables the authentication provider required for access, uses role-based access control (RBAC) to manage authorization, and defines four types of users: active users, administrative users, current users, and suspended users, and five types of roles: Public, Reader, Editor, Publisher, Architect, and Admin. At the same time, this application also formulates a suitable backup and recovery strategy based on the deployment environment of Neo4j, and regularly stores database backups online or offline to an off-site server different from the database file. The first backup uses a full backup, and subsequent backups use differential backups. This application also enables a logging program for continuous analysis or specific investigations, especially the recording of security event logs and query logs.

[0137] The present application proposes to realize the automatic construction, storage and visualization of graphs based on Neo4j graph database, Py2neo library and Neo4j Bloom, wherein "Neo4j graph database", "Py2neo library" and "Neo4j Bloom" can be replaced by any graph database, script library and graph visualization tool with the same functions. The present application proposes to realize complex association relationship query based on Cypher query language, wherein "Cypher query language" can be replaced by any graph database query language adapted to the selected graph database. The present application also proposes a graph database authentication and authorization strategy based on RBAC, four types of users and five types of roles, and a graph database backup and recovery strategy combining online and offline, complete backup and differential backup, wherein "graph database authentication and authorization strategy based on RBAC, four types of users and five types of roles" and "graph database backup and recovery strategy combining online and offline, complete backup and differential backup" can be replaced by any graph database management strategy with the same functions and suitable.

[0138] This application proposes a graph automation construction, storage and visualization solution based on Neo4j graph database, Py2neo library and Neo4j Bloom, and a graph database management solution for authentication and authorization, backup and recovery of Neo4j, so as to quickly respond to complex relationship queries and support all-round and multi-dimensional graph visualization, and effectively improve the overall security and stability of Neo4j.

[0139] As an optional implementation of the present application, Fig. 20 As shown, S600 includes:

[0140] S610, converting the positive risk knowledge triples and the randomly generated negative risk knowledge triples contained in the risk knowledge graph of the big data platform from the Euclidean space to the hyperbolic space, and obtaining a low-dimensional embedding representation of all entities and the relationships between them;

[0141] S620, using all low-dimensional embedding representations to train the graph attention network, scoring each risk knowledge triplet during the training process, and distinguishing correct and incorrect risk knowledge triplet based on the score;

[0142] The risk knowledge graph of the big data platform is transferred from the Euclidean space to the hyperbolic space with constant negative curvature for representation learning. The hyperbolic space can better simulate the non-Euclidean geometric properties in the data due to its advantages in processing data with complex network structures. Using the graph attention network, in the neighborhood of the entity, a dual-level attention mechanism based on entities and relationships is used to aggregate neighborhood feature information and embed the entity into a low-dimensional hyperbolic space. Entity-level attention focuses on the impact of different entities on a given entity, and relationship-level attention focuses on the impact of the relationship between entities on link prediction. As shown in formula (1), the score function designed based on the characteristics of the hyperbolic space is used to calculate the score of each triple to predict the possible links (possible relationships between entities) in the risk graph. Triples with high scores are more likely to be real links.

[0143] This step uses a scoring function to score each risk knowledge triplet, and the scoring function is expressed as:

[0144]

[0145] In the formula, are the hyperbolic embedded representations of the head entity and the tail entity, respectively. is the relation parameter vector, is a diagonal relationship matrix, the bias term b h and b t is a hyperbolic decision boundary, which determines the h The radius of the hypersphere centered at v tThe radius of fall is When the triple (v h ,r,v t ) is established;

[0146] S630, taking the correct risk knowledge triple representation as a positive sample and taking the incorrect risk knowledge triple representation as a negative sample, repeatedly iteratively training the preset enhancement model until the contrast loss function of the enhancement model remains unchanged, thereby obtaining a trained enhancement model;

[0147] Based on the low-dimensional embedding representation of entities in hyperbolic space, the contrastive learning method is introduced to enhance the model's ability to distinguish correct and incorrect triplets. Contrastive learning optimizes the embedding representation of entities and relationships by comparing positive links (known entity relationships) and negative links (randomly generated entity relationships), thereby improving the accuracy of link prediction. As shown in formula (2), the contrastive loss function (InfoNCE loss) is used to optimize the model, and the model's discrimination ability is improved by maximizing the similarity of positive samples and minimizing the similarity of negative samples. Among them, φ() represents the score function of the triplet, and |N| represents the set of negative samples. The temperature parameter τ is used to adjust the relative importance of negative samples in the loss function, and different τ values ​​will affect the performance and effect of the model. A smaller τ value can make the loss function pay more attention to difficult negative samples, thereby improving the robustness and generalization ability of the model. However, when the τ value is too small, the model may overfit the label noise in the training data, resulting in a decrease in the generalization ability of the model. The performance evaluation indicators of the model include mean reciprocal rank (MRR) and Hits@N.

[0148] The contrast loss function is expressed as:

[0149]

[0150] In the formula, N represents the negative sample set, h represents the head entity vector of the positive sample, t represents the tail entity vector of the positive sample, T represents the positive sample set, and t i ′ Represents the tail entity vector of the negative sample, φ(h,r,t i ′ ) represents the triple (h, r, t i ′ ), φ(h,r,t) represents the score of the triple (h,r,t), and τ represents the temperature parameter. φ() represents the scoring function of the triple.

[0151] S640: Use the trained enhancement model to complete the knowledge of the risk knowledge graph of the big data platform.

[0152] This application proposes knowledge completion based on hyperbolic graph attention network and contrastive learning, wherein the hyperbolic graph attention network and the enhanced model can be replaced by any mainstream method with the same knowledge graph completion function, and the score function of the triplet and the loss function of the model can also be replaced by any mainstream function with the same function. In addition, the construction method of positive and negative samples in contrastive learning can also be replaced by any mainstream method.

[0153] After knowledge completion, cybersecurity experts will conduct a final assessment of the quality of the risk knowledge graph of the big data platform to verify the effectiveness of the constructed risk graph in actual security scenario applications.

[0154] This application proposes a knowledge completion solution that combines a hyperbolic graph attention network and contrastive learning. The hyperbolic graph attention network can dynamically focus on important nodes and edges in the graph through the attention mechanism, so that the model can more accurately understand and abstract the complex interactions between entities and relationships. Contrastive learning helps the model better capture the key factors that determine whether a relationship exists or not by analyzing the differences between positive and negative samples. The combination of the two can greatly explore various hidden associations that may exist in the graph.

[0155] As an optional implementation of the present application, refer to Fig.21 , S700 includes:

[0156] S710, deploying the knowledge processing method in S300 to S600 based on the stream processing engine, starting the stream processing engine to monitor the risk knowledge base of the big data platform in real time, and using the stream processing technology to map the real-time changes in the risk knowledge base of the big data platform to the corresponding part of the risk knowledge graph of the big data platform, so as to automatically update the risk knowledge graph of the big data platform;

[0157] The above-mentioned processing methods of knowledge extraction, knowledge fusion, knowledge storage, and knowledge completion are deployed based on the stream processing engine in the big data scenario, and the risk knowledge base of the big data platform is monitored in real time. Stream processing technology (such as Apache Kafka or Apache Flink) is used to map the real-time changes in the knowledge base to the corresponding parts of the risk map, such as adding new entities, updating attributes, deleting outdated information, etc., to achieve dynamic updating of the risk map.

[0158] This application proposes to use stream processing technology (such as Apache Kafka or Apache Flink) to map real-time changes in the knowledge base, where "stream processing technology" can be replaced by any stream processing method with the same function.

[0159] S720, according to the query results and feedback of the platform administrator, correct the errors in the big data platform risk knowledge graph or supplement the missing information to manually update the big data platform risk knowledge graph.

[0160] This step establishes a knowledge feedback mechanism to promptly correct errors in the graph or supplement missing information in the graph based on the query results and feedback of the platform manager. Create knowledge graph snapshots regularly and manage different versions of the risk graph so that it can be quickly restored to the previous stable state when an error occurs.

[0161] This application proposes a knowledge graph dynamic update solution based on the deployment of a stream processing engine. The stream processing engine can automatically handle the extraction, fusion and update of risk knowledge, greatly improving the maintenance efficiency of the knowledge graph and reflecting the latest risk changes in real time. At the same time, the scalability of stream processing also ensures that the knowledge graph system can adapt to the growth of data volume and maintain efficient processing capabilities regardless of the data size.

[0162] It is worth noting that the terms "first" and "second" in this application are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0163] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components or steps.

[0164] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.

Claims

1. A method for constructing a risk knowledge graph for a big data platform, characterized in that: include: S100, collect relevant risk knowledge of the big data platform and organize and clean the data to build a risk knowledge base of the big data platform; The big data platform risk knowledge base includes structured risk knowledge and unstructured risk knowledge; S200, adopts the method of combining manual definition with domain expert knowledge to build the ontology of asset, data and business risk graphs from bottom to top, and obtains the ontology of risk knowledge graph of big data platform; S300, based on the definition of the ontology in the risk knowledge graph of the big data platform, and using the knowledge extraction model to extract the entities and the relationships between the entities in the risk knowledge graph of the big data platform, and pair the entities with the risk attributes based on the risk knowledge base of the big data platform to form risk knowledge triples; The risk knowledge triples include entities, relationships between entities and entity attributes; S400, extracting the features of each entity in the risk knowledge triple, and using the features to calculate the similarity between the entities, using the similarity to determine matching entities, and fusing the matching entities to re-form the risk knowledge triple; S500 stores the reorganized risk knowledge triples and automatically constructs them into a risk knowledge graph on the big data platform; S600, completing the knowledge of the big data platform risk knowledge graph, and performing quality assessment on the knowledge-completed big data platform risk knowledge graph; S700 dynamically updates the risk knowledge graph of the knowledge completion big data platform to correct and supplement missing information.

2. The method for constructing a risk knowledge graph for a big data platform according to claim 1, characterized in that: S100 includes: S110, collecting data information and all possible sensing means for sensing the data information from the big data platform, wherein the data information includes: assets, data, and business; S120, using a penetration testing tool set that aggregates all possible perception means to automatically and actively perceive the data information, and combining expert knowledge to supplement the risk knowledge that cannot be perceived in the data information to obtain risk knowledge; the risk knowledge includes: structured risk knowledge and unstructured risk knowledge; S130, performing data cleaning operations on the risk knowledge and forming a risk knowledge base of the big data platform.

3. The method for constructing a risk knowledge graph for a big data platform according to claim 1, characterized in that: S200 includes: S210, constructing the ontology of the asset risk map from the bottom up by combining manual definition with domain expert knowledge, wherein the asset risk map uses the asset as an ontology and defines the association relationship and risk attributes between the ontologies; S220, using manual definition combined with domain expert knowledge to construct the ontology of the data risk map from the bottom up, the data risk map is divided into six risk sub-graphs of data collection, data storage, data transmission, data processing, data sharing, data backup, recovery and deletion according to the full life cycle of the data on the big data platform, and the ontology and risk attributes of each risk sub-graph are defined; S230, using manual definition combined with domain expert knowledge to construct the ontology of the business risk map from bottom to top, wherein the business risk map describes the association between businesses and between businesses and data, and defines various ontologies and risk attributes; S240, performing quality assessment on each ontology in the asset risk map, the data risk map and the business risk map to obtain assessment results of each ontology.

4. The method for constructing a risk knowledge graph for a big data platform according to claim 1, characterized in that: S300 includes: S310, using the unstructured risk knowledge in the risk knowledge base of the big data platform as risk corpus, manually annotating the risk corpus with the help of LabelStudio open source data annotation tool according to the definition of the ontology in the risk knowledge graph of the big data platform, and annotating the annotated risk corpus based on the BIO ternary annotation rule to obtain the annotated risk corpus; S320, inputting the annotated risk corpus into the BERT submodule included in the knowledge extraction model to perform dynamic vector representation to obtain encoded first data; and inputting the first data into the BiLSTM submodule included in the knowledge extraction model to learn long-distance text information, and then using CRF to obtain the global optimal annotation sequence, so as to output the entities contained in the risk corpus; S330, dividing the entity into a subject and an object, and inputting the entity into a PERT submodule together with the risk corpus to extract context features of the relational data to obtain encoded second data, and inputting the second data into an FC submodule to obtain a first risk knowledge triple containing entities and risk relations between entities; S340, based on the location and context information of the entity, the entity in the risk knowledge triplet is matched with the risk attribute in the risk knowledge base of the big data platform by using rule matching to form a second risk knowledge triplet including the entity and its attributes.

5. The method for constructing a risk knowledge graph for a big data platform according to claim 4, characterized in that: The knowledge extraction model is implemented based on the BERT-BiLSTM-CRF model and consists of two parts: named entity recognition and relationship extraction. The named entity recognition part includes three sub-modules: BERT, BiLSTM, and CRF, and the relationship extraction part includes two sub-modules: PERT and FC. The BERT sub-module is fine-tuned by integrating the open source network threat intelligence knowledge graph. The risk knowledge triple includes a first risk knowledge triple and a second risk knowledge triple. The first risk knowledge triple is expressed in the form of G xry =(entity x, relation r, entity y), G xry Indicates that there is a relationship r between entity x and entity y. The second risk knowledge triple is expressed as G xpv =(entity x, risk attribute p, attribute value v), G xpv It means that entity x has a certain risk attribute p and its attribute value is v.

6. The method for constructing a risk knowledge graph for a big data platform according to claim 1, characterized in that: S400 includes: S410, deduplicating the risk knowledge triples, extracting features of entities in the deduplicated risk knowledge triples, and combining related features of the entities to form all features of the entities; S420, for all features of the entity, using TF-IDF weighted cosine similarity as a metric, calculating similarities between entities, determining entities with similarities greater than a similarity threshold, and taking them as matched entities; S430, unifying the matched entities into the same entity name, merging the entity attributes, and obtaining a risk knowledge triple after matching processing; S440, de-duplicate the risk knowledge triples after matching, and perform quality assessment to obtain an assessment result.

7. The method for constructing a risk knowledge graph for a big data platform according to claim 1, characterized in that: S500 includes: S510, storing the reorganized risk knowledge triples based on the Neo4j graph database; S520, using the Py2neo library to convert the reconstructed risk knowledge triples into nodes and edges in the risk knowledge graph of the big data platform, so as to automatically construct the risk knowledge graph of the big data platform; wherein the entities in the risk knowledge triples are mapped to nodes in the risk knowledge graph of the big data platform, the risk relationships between the entities are mapped to edges in the risk knowledge graph of the big data platform, and the risk attributes are mapped to attributes of the nodes; S530, using a visualization tool Neo4j Bloom, visually display the risk knowledge graph of the big data platform.

8. The method for constructing a risk knowledge graph for a big data platform according to claim 1, characterized in that: S600 includes: S610, converting the positive risk knowledge triples and the randomly generated negative risk knowledge triples contained in the risk knowledge graph of the big data platform from the Euclidean space to the hyperbolic space, and obtaining a low-dimensional embedding representation of all entities and the relationships between them; S620, using all low-dimensional embedding representations to train the graph attention network, scoring each risk knowledge triplet during the training process, and distinguishing correct and incorrect risk knowledge triplet based on the score; S630, taking the correct risk knowledge triple representation as a positive sample and taking the incorrect risk knowledge triple representation as a negative sample, repeatedly iteratively training the preset enhancement model until the contrast loss function of the enhancement model remains unchanged, thereby obtaining a trained enhancement model; S640: Use the trained enhancement model to complete the knowledge of the risk knowledge graph of the big data platform.

9. The method for constructing a risk knowledge graph for a big data platform according to claim 8, characterized in that: Scoring each risk knowledge triple in S620 includes: Each risk knowledge triple is scored using a scoring function, which is expressed as: In the formula, are the hyperbolic embedded representations of the head entity and the tail entity, respectively. is the relation parameter vector, is a diagonal relation matrix, the bias term b h and b t is a hyperbolic decision boundary, which determines the h The radius of the hypersphere centered at v t The radius of fall is When the triple (v h ,r,v t ) is established; The contrast loss function in S630 is expressed as: In the formula, N represents the negative sample set, h represents the head entity vector of the positive sample, t represents the tail entity vector of the positive sample, T represents the positive sample set, and t i ′ Represents the tail entity vector of the negative sample, φ(h,r,t i ′ ) represents the triple (h, r, t i ′ ), φ(h,r,t) represents the score of the triple (h,r,t), and τ represents the temperature parameter. φ() represents the scoring function of the triple.

10. The method for constructing a risk knowledge graph for a big data platform according to claim 1, characterized in that: S700 includes: S710, deploying the knowledge processing method in S300 to S600 based on the stream processing engine, starting the stream processing engine to monitor the risk knowledge base of the big data platform in real time, and using the stream processing technology to map the real-time changes in the risk knowledge base of the big data platform to the corresponding part of the risk knowledge graph of the big data platform, so as to automatically update the risk knowledge graph of the big data platform; S720, according to the query results and feedback of the platform administrator, correct the errors in the big data platform risk knowledge graph or supplement the missing information to manually update the big data platform risk knowledge graph.

Images