Self-improvement attack and defense integrated data lake and warehouse system and method and storage medium

Abstract

The invention relates to a self-improvement attack and defense integrated data lake warehouse system and method and a storage medium, and the system comprises a data lake warehouse which is used for storing multi-mode network security data; the network security data comprises standard structured data written by the model communication and control service; the pre-trained network security large model is used for initiating a data query request to the model communication and control service, so that the model communication and control service executes a data query operation on the data lake bin according to the data query request and returns a query result; analyzing and reasoning according to a query result returned by the model communication and control service to generate insight data; and the model communication and control service is used for generating standard structured data based on the insight data and writing the standard structured data into the data lake bin. According to the method and the device, the problem that the data lake and warehouse system cannot dynamically and automatically perfect data in the prior art is solved, and the dynamic self-perfection function of the data lake and warehouse system is realized.

Description

Technical Field

[0001] This application relates to the field of cybersecurity, and in particular to a self-sustaining, integrated offensive and defensive data lake warehouse system, method, and storage medium. Background Technology

[0002] With the rapid development of information technology, network systems and online services have become deeply integrated into people's daily work and lives. These systems and service platforms typically need to process massive amounts of core business data and user privacy data. However, the ever-changing and escalating threats of cyberattacks, malicious behaviors, and other abnormal events pose serious challenges to their stability and security. Therefore, an intelligent, efficient, self-improving, and integrated offensive and defensive data lake warehouse technology is of paramount importance for ensuring the stability and security of such systems.

[0003] Existing technologies typically rely on data lake warehouse architectures, utilizing large-scale network security models to perform in-depth analysis, threat detection, and incident response on the multi-source, heterogeneous network security data stored within the data lake warehouse. The drawback of these existing technologies is that traditional data lake warehouse systems are essentially passive data repositories; data is fixed after being written, and its value primarily lies in passive querying and analysis. The system itself does not actively generate new data to enhance itself. Over time, without continuous human analysis and enrichment, the timeliness and value of the data gradually diminish.

[0004] There is currently no effective solution to the problem that data lake warehouse systems in related technologies cannot dynamically and self-improve data. Summary of the Invention

[0005] This embodiment provides a self-improving, integrated offensive and defensive data lake warehouse system, method, and storage medium to solve the problem in related technologies that data lake warehouse systems cannot dynamically improve their data.

[0006] Firstly, this embodiment provides a self-improving, integrated offensive and defensive data lake warehouse system, including:

[0007] The data lake warehouse, connected to a pre-built model communication and control service, is used to store multimodal cybersecurity data; the cybersecurity data includes standard structured data written by the model communication and control service.

[0008] A pre-trained cybersecurity big data model is connected to the model communication and control service to initiate data query requests to the model communication and control service. This enables the model communication and control service to perform data query operations on the data lake warehouse based on the data query requests and return the query results. The model communication and control service then analyzes and infers based on the query results returned by the model communication and control service to generate insightful data.

[0009] The model communication and control service generates the standard structured data based on the insight data and writes the standard structured data into the data lake warehouse.

[0010] In some embodiments, the model communication and control service includes a bidirectional communication interface;

[0011] The bidirectional communication interface is connected to the data lake warehouse and the network security big model respectively, and is used to convert the data query request of the network security big model into a structured query of the data lake warehouse, and return the query result to the network security big model in natural language or structured format.

[0012] In some embodiments, the model communication and control service includes a toolset and an execution engine;

[0013] The toolset and execution engine are connected to the cybersecurity big model and are used to provide one or more executable security operation tools for the cybersecurity big model.

[0014] In some embodiments, the security operation tools include port scanning tools, asset information query tools, domain name reputation query tools, or attack graph generation tools;

[0015] The port scanning tool is connected to the network security big model and is used to probe specified network addresses and ports;

[0016] The asset information query tool is connected to the network security big model and is used to retrieve the corresponding hardware configuration information based on the asset identifier.

[0017] The domain name reputation query tool is connected to the network security big data model and is used to query the reputation assessment results based on a specified domain name.

[0018] The attack graph generation tool is connected to the network security big model and is used to construct a structured attack graph describing the attack steps.

[0019] In some embodiments, the model communication and control service includes a data structuring and verification module;

[0020] The data structuring and verification module is connected to the network security big model and is used to convert the insight data generated by the network security big model into standard structured data that conforms to a predefined data pattern.

[0021] In some of these embodiments, the insight data includes defensive insight data and offensive insight data;

[0022] The defensive insight data includes threat indicator types, threat indicator values, confidence levels, and attack stages;

[0023] The attack insight data includes a structured attack graph that describes the attack steps.

[0024] Secondly, this embodiment provides a self-improving, integrated attack and defense data lake warehouse method. This method is applicable to the self-improving, integrated attack and defense data lake warehouse system described in the first aspect above. The method includes:

[0025] The pre-trained cybersecurity big data model initiates a data query request to the pre-built model communication and control service;

[0026] In the model communication and control service, a data query operation is performed on the data lake warehouse according to the data query request, and the query result is returned to the network security big model;

[0027] Within the aforementioned cybersecurity big data model, the query results are analyzed and reasoned to generate insightful data.

[0028] In the model communication and control service, standard structured data is generated based on the insight data, and the standard structured data is written into the data lake warehouse.

[0029] In some embodiments, the cybersecurity big data model analyzes and infers based on the query results to generate insight data, and further includes:

[0030] If it is the defensive analysis phase, the network security big model identifies potential security threats based on the query results and generates defensive insight data that includes threat indicator types, threat indicator values, confidence levels, and attack phases.

[0031] If it is the attack analysis phase, the network security big model simulates attacker behavior based on the query results and generates attack insight data containing a structured attack graph.

[0032] In some embodiments, generating standard structured data and writing the standard structured data into the data lake warehouse includes:

[0033] In the model communication and control service, the correctness of the standard structured data is verified based on a predefined data format;

[0034] If the standard structured data conforms to the predefined data format, the verification is successful, and the standard structured data is written into the data lake warehouse.

[0035] If the standard structured data does not conform to the predefined data format, the verification fails, the data write operation is blocked, and an error log is generated.

[0036] Thirdly, this embodiment provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the self-improving, integrated offensive and defensive data lake warehouse method described in the second aspect above.

[0037] Compared to related technologies, the self-improving, integrated offensive and defensive data lake warehouse system, method, and storage medium provided in this embodiment connect to a pre-built model communication and control service through the data lake warehouse to store multimodal cybersecurity data. The cybersecurity data includes standard structured data written by the model communication and control service. A pre-trained cybersecurity big model connects to the model communication and control service to initiate data query requests, enabling the model communication and control service to perform data query operations on the data lake warehouse based on the data query requests and return the query results. Analysis and reasoning are performed based on the query results returned by the model communication and control service to generate insight data. Based on the insight data, the model communication and control service generates standard structured data and writes it into the data lake warehouse. By introducing the model communication and control service as an intelligent intermediary, the collaborative cooperation between the data lake warehouse and the cybersecurity big model is achieved, enabling the system to continuously evolve during use. This solves the problem in related technologies where data lake warehouse systems cannot dynamically self-improve data, realizing the dynamic self-improvement function of the data lake warehouse system.

[0038] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a hardware structure block diagram of the terminal device of a self-improving, integrated offensive and defensive data lake warehouse system provided in one embodiment of this application;

[0041] Figure 2 This is a structural block diagram of a self-improving, integrated offensive and defensive data lake warehouse system provided in one embodiment of this application;

[0042] Figure 3 This is a structural block diagram of a model communication and control service provided in an embodiment of this application;

[0043] Figure 4This is a structural block diagram of the toolset and execution engine provided in one embodiment of this application;

[0044] Figure 5 This is a flowchart of a self-improving, integrated offensive and defensive data lake warehouse method provided in one embodiment of this application;

[0045] Figure 6 This is a flowchart illustrating a self-improving, integrated offensive and defensive data lake warehouse method provided in one embodiment of this application.

[0046] In the diagram: 102, Processor; 104, Memory; 106, Transmission Device; 108, Input / Output Device; 200, Data Lake Warehouse; 300, Network Security Large Model; 400, Model Communication and Control Service; 410, Bidirectional Communication Interface; 420, Toolset and Execution Engine; 430, Data Structuring and Verification Module; 421, Port Scanning Tool; 422, Asset Information Query Tool; 423, Domain Name Reputation Query Tool; 424, Attack Graph Generation Tool. Detailed Implementation

[0047] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0049] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal of the self-improving, integrated attack and defense data lake warehouse method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0050] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the self-improving integrated attack and defense data lake warehouse method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0051] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0052] This embodiment provides a self-improving, integrated offensive and defensive data lake warehouse system. Figure 2 This is a block diagram of the self-improving, integrated offensive and defensive data lake warehouse system in this embodiment, as shown below. Figure 2 As shown:

[0053] The self-sufficient, integrated offensive and defensive data lake warehouse system includes Data Lake Warehouse 200, Network Security Big Model 300, and Model Communication and Control Service 400;

[0054] The data lake warehouse 200 is connected to the pre-built model communication and control service 400 for storing multimodal cybersecurity data; the cybersecurity data includes standard structured data written by the model communication and control service 400.

[0055] The pre-trained cybersecurity big data model 300 is connected to the model communication and control service 400 to initiate data query requests to the model communication and control service 400. This enables the model communication and control service 400 to perform data query operations on the data lake warehouse 200 based on the data query requests and return the query results. The model communication and control service 400 then analyzes and infers based on the query results returned by the model communication and control service 400 to generate insightful data.

[0056] Model communication and control service 400 generates standard structured data based on insight data and writes the standard structured data into data lake warehouse 200.

[0057] Specifically, the Data Lake Warehouse 200, serving as the system's unified data foundation, is configured to store multimodal security data, including structured logs and semi-structured data from traditional security devices, as well as unstructured text and structured threat intelligence generated from large models. There are no restrictions on the data types stored in the Data Lake Warehouse. The Data Lake Warehouse possesses ACID (Atomicity, Consistency, Isolation, Durability) transaction capabilities and schema enforcement mechanisms to ensure the atomicity and consistency of data writes.

[0058] The Cybersecurity Big Model 300 can be an AI-powered large language model that has undergone further pre-training and fine-tuning with instructions on a large-scale corpus of cybersecurity data; or it can be a multimodal model specifically designed for cybersecurity analysis. There are no restrictions on the type of cybersecurity big model. The Cybersecurity Big Model 300 is capable of understanding complex cybersecurity scenarios and autonomously deciding on subsequent actions based on the analysis objectives, rather than being limited to passively answering queries.

[0059] Model Communication and Control Service 400 is an intelligent middleware that integrates tool execution, state management, intent understanding, and, most importantly, data formatting and validation. It forces the transformation of the model's free, unstructured output into standardized, reliable structured data. This includes, but is not limited to, a bidirectional communication interface, a toolset and execution engine, and a data structuring and validation module; however, no restrictions are placed on the types of components that make up the Model Communication and Control Service.

[0060] Through the above steps, the system generates insight data through autonomous detection of the network security big data model, and uses model communication and control services to convert the unstructured insight data generated by the big data model into standard structured data and write it back to the data lake warehouse. This solves the problem that data lake warehouse systems cannot dynamically improve data in related technologies, effectively overcomes the inherent defects of existing technologies such as static value of secure data lake warehouses and inability to self-evolve, and realizes the dynamic enhancement capability of the system knowledge base to continuously and autonomously improve in interaction.

[0061] The following is a detailed description of each of the above components:

[0062] In some embodiments, the model communication and control service includes a bidirectional communication interface;

[0063] The bidirectional communication interface 410 connects to the data lake warehouse and the network security big model respectively. It is used to convert the data query requests of the network security big model into structured queries to the data lake warehouse and return the query results to the network security big model in natural language or structured format.

[0064] Specifically, such as Figure 3 As shown, the bidirectional communication interface 410 implements the conversion of semantics in the network security domain into data manipulation instructions. When the network security big data model initiates a request in natural language, this interface parses the entity device, time range, and query target, and maps them to the corresponding data tables, fields, and filtering conditions in the data lake warehouse. Subsequently, based on the mapping relationship, an executable structured query is automatically generated, which can be a specific SQL query statement or a Spark job; there are no restrictions on the structured query method. After the query is executed, the bidirectional communication interface returns the query results to the network security big data model in natural language or structured format.

[0065] This embodiment solves the semantic recognition problem between the large-scale network security model and the underlying data system. It transforms analysts' natural language into a structured query language, lowering the technical barrier to data access and analysis in security operations and providing reliable input data for subsequent analysis and reasoning.

[0066] In some embodiments, the model communication and control service includes a toolset and an execution engine;

[0067] Toolset and Execution Engine 420, connected to the cybersecurity big model, provides one or more executable security operation tools for the cybersecurity big model.

[0068] Specifically, the toolset encapsulates a variety of security operation tools. When the cybersecurity big data model needs to perform proactive probing to obtain data outside the context based on inference, it can select and invoke the appropriate tool from the toolset. The execution engine, as a secure and controllable runtime environment, is responsible for receiving the invocation instructions, executing the tool within preset policy boundaries (such as target scope, concurrency limits, and resource quotas), and capturing its raw output. Subsequently, the execution engine cleans and formats the raw output and feeds it back to the cybersecurity big data model as new, model-understandable data for the next round of inference and decision-making.

[0069] This embodiment addresses the limitation of large-scale cybersecurity models being able to analyze only existing data. The new data generated by the model through tool execution can be analyzed, transformed into new knowledge, and backfilled into the data lake repository. This enhances the system's proactive defense and threat detection capabilities, providing support for a security paradigm that moves from static analysis to dynamic interaction.

[0070] In some of these embodiments, the security operation tools include port scanning tools, asset information query tools, domain reputation query tools, or attack graph generation tools;

[0071] Port scanning tool 421 connects to a network security big model and is used to probe specified network addresses and ports;

[0072] Asset Information Query Tool 422 is connected to the network security big model and is used to retrieve corresponding hardware configuration information based on asset identifiers.

[0073] Domain reputation query tool 423 is connected to the network security big model and is used to query reputation assessment results for a specified domain name.

[0074] Attack graph generation tool 424, connected to a large network security model, is used to construct structured attack graphs that describe attack steps.

[0075] Specifically, such as Figure 4 As shown, the port scanning tool responds to the call command of the network security big data model, performs port connectivity detection on the target IP address, identifies open ports and their corresponding services and version information, and feeds back the detection results in a structured manner to the model as an important basis for judging host risk.

[0076] The asset information query tool retrieves and returns detailed technical attributes of the asset from the configuration management library of the data lake warehouse based on the asset identifier (such as hostname, IP or asset ID) provided by the model. These attributes include, but are not limited to, operating system type and version, list of installed software, hardware configuration, network interface information and security domain. This provides key information for the model to assess the scope of vulnerability impact and attack feasibility. There are no restrictions on the type of technical attributes.

[0077] The domain reputation query tool queries multiple internal reputation databases simultaneously based on the domain name or IP address specified by the model to obtain the entity's historical malicious activity records, associated malware families, reputation scores, and threat type tags. The comprehensive evaluation results are returned in a structured form to help the model determine whether a network entity is malicious infrastructure.

[0078] The attack graph generation tool uses the initial access points and security elements provided by the model to automatically deduce the steps that attackers may take, such as lateral movement and privilege escalation, using a graph computing engine to generate a structured attack graph. This graph intuitively reveals the system's vulnerable chains and potential attack paths, and outputs the graph as a computable and storable data object.

[0079] This embodiment provides a standardized and scalable set of proactive security operational capabilities, enabling the system to proactively verify hypotheses and discover unknown threats. Simultaneously, the new data generated by the tool execution can be transformed into new threat indicators, asset profiles, or attack pattern knowledge, and written back to the data lake warehouse in a standardized manner, thereby realizing a shift from passive analysis to proactive interaction, and from data consumption to data creation.

[0080] In some embodiments, the model communication and control service includes a data structuring and verification module;

[0081] The data structuring and validation module 430 is connected to the cybersecurity big model and is used to convert the insight data generated by the cybersecurity big model into standard structured data that conforms to a predefined data pattern.

[0082] Specifically, when the cybersecurity big data model generates new security insights, the data structuring and verification module intervenes to transform unstructured natural language descriptions into standard structured data containing complete attribute fields (such as threat indicator type, specific value, confidence level, attack stage, and attribution information). Standard structured data can be a JSON (JavaScript Object Notation) object conforming to the Structured Threat Information Expression (STIX) standard or a structured data object conforming to the Open Indicator of Compromise (OpenIOC) standard. Attribute fields can include threat indicator type, specific value, confidence level, or attack stage, etc. There are no restrictions on the type of standard structured data or the type of attribute fields.

[0083] In this embodiment, the data structuring and validation module solves the problem of reliably converting the free and unstructured output of large models into standard structured data that conforms to a predefined data pattern. Through standardized templates and a mandatory validation process, it ensures that the new data generated and stored during the system's self-improvement process are machine-readable, of controllable quality, and consistent with standards.

[0084] In some of these embodiments, the insight data includes defensive insight data and offensive insight data;

[0085] Defensive insights data includes threat indicator types, threat indicator values, confidence levels, and attack phases;

[0086] The offensive insights data includes structured attack graphs that describe the attack steps.

[0087] Specifically, defensive insight data refers to the structured conclusions formed by the cybersecurity big data model when performing defensive tasks such as threat detection, incident response, or abnormal behavior analysis. These conclusions are generated by performing correlation analysis and reasoning on multi-source heterogeneous security data in the data lake warehouse, and can guide specific defensive actions. The data includes threat indicator types, threat indicator values, confidence levels, and attack stages. There are no restrictions on the types of data included in defensive insight data.

[0088] Attack insight data is the result of a large-scale cybersecurity model simulating an attacker's perspective and performing attack path simulations. Attack insight data is a structured attack graph, where nodes represent the state of system assets (hosts, users, data) at different stages of the attack, and edges represent attack actions (such as vulnerability exploitation and lateral movement) and preconditions. This graph reveals the complete vulnerability chain from the initial intrusion point to critical assets, and marks the specific vulnerabilities exploited, the necessary privileges, and potential detection points.

[0089] This embodiment classifies and structures the defensive and offensive aspects of insight data. These two types of data form a self-reinforcing feedback loop in the system: new threat indicators discovered by defensive analysis can automatically trigger an offensive simulation to verify their potential harmful paths; while the attack graph generated by the offensive simulation, which reveals key vulnerabilities and attack paths, will serve as high-priority defensive work orders to guide the implementation of targeted protection.

[0090] This embodiment also provides a self-improving, integrated attack and defense data lake warehouse method, which is applicable to any of the self-improving, integrated attack and defense data lake warehouse systems described in the above embodiments. Figure 5 This is a flowchart of the self-improving, integrated attack and defense data lake warehouse method in this embodiment, as follows: Figure 5 As shown, the process includes the following steps:

[0091] Step S210: Initiate a data query request to the pre-built model communication and control service through the pre-trained network security big model;

[0092] Step S220: In the model communication and control service, a data query operation is performed on the data lake warehouse according to the data query request, and the query result is returned to the network security big model; in the network security big model, analysis and reasoning are performed based on the query result to generate insight data;

[0093] Step S230: In the model communication and control service, standard structured data is generated based on the insight data, and the standard structured data is written into the data lake warehouse.

[0094] Specifically, such as Figure 5 As shown, firstly, after receiving a security analysis task, the pre-trained cybersecurity big data model interacts with the model communication and control service through natural language and initiates a data query request to the model communication and control service.

[0095] Secondly, the bidirectional communication interface in the model communication and control service instantly captures the request. Through intent recognition and semantic parsing techniques, it decomposes the request into structured elements and maps them to corresponding data tables, fields, and filtering conditions in the data lake warehouse. These elements are then automatically assembled into an executable, specific query statement. The model communication and control service submits the assembled structured query to the data lake warehouse for execution, and the data lake warehouse returns the original query result set. The model communication and control service then performs preliminary aggregation, summarization, or key information extraction on the result set, and finally feeds it back to the cybersecurity big data model in a format that facilitates further reasoning.

[0096] Finally, based on the returned context data, the cybersecurity big data model utilizes its built-in cybersecurity knowledge (such as attack patterns and abnormal behavior baselines) to perform deep correlation, causal reasoning, and threat assessment, ultimately generating unstructured insight data. The data structuring module of the model communication and control service proactively intervenes, guiding the cybersecurity big data model to populate the aforementioned unstructured threat assessment according to a pre-selected standardized template, generating a machine-readable, standardized structured data object containing complete fields. The syntax compliance and logical consistency of this data object are automatically validated. Upon successful validation, this standardized structured data is persistently written as a new, high-confidence threat intelligence record to the designated security intelligence table in the data lake repository. This newly added record can immediately be queried and referenced by all subsequent analysis tasks, detection rules, or response scripts.

[0097] This embodiment transforms valuable new insights (new threat indicators) generated during model analysis into high-quality, reusable data assets within the system itself. This transforms the data lake warehouse from a static architecture that merely stores historical input data into a dynamic system capable of continuously generating new knowledge through its own analysis, thereby achieving continuous and autonomous enhancement of its internal security data system. This effectively solves the problem of static value in traditional security data platforms.

[0098] In some embodiments, the self-improving, integrated offensive and defensive data lake warehouse method further includes the following steps:

[0099] If it is the defensive analysis phase, the cybersecurity big data model identifies potential security threats based on the query results and generates defensive insight data that includes threat indicator types, threat indicator values, confidence levels, and attack phases.

[0100] In the attack analysis phase, the cybersecurity big data model simulates attacker behavior based on the query results, generating attack insight data containing a structured attack graph.

[0101] Specifically, in the defensive analysis phase, the cybersecurity big data model acquires multi-source security data related to the mission objectives from the data lake warehouse through model communication and control services. Based on its built-in cybersecurity knowledge base, it performs contextual analysis, pattern matching, and causal reasoning on the data to identify potential security threats. During this process, the model transforms its analytical conclusions into a structured output containing a specific set of attributes. This structured output is defined as defensive insight data, and its core attributes include at least the threat indicator type used to identify the threat category, the threat indicator value representing a specific instance of the threat, a confidence assessment reflecting the reliability of the judgment, and the attack stage the threat is in within the attack lifecycle. The data types included in the defensive insight data are not limited here.

[0102] During the attack analysis phase, the cybersecurity big data model proactively identifies unexploited potential risk paths within the system. Based on high-value clues output from the defensive analysis phase, the model extensively queries the network topology, asset vulnerability database, access control policies, and account permission tables in the data lake warehouse through model communication and control services. Using this information, the model leverages its built-in attack tactics knowledge base to simulate how attackers exploit initial vulnerabilities to gain a foothold, how they move laterally across network segments, and ultimately how they escalate privileges to achieve their attack goals. The simulation results are constructed into a structured attack graph, clearly depicting each attack action, dependency, exploited vulnerability, and affected assets in a machine-readable format.

[0103] This embodiment clearly defines and implements the connection and collaboration between defensive and offensive analysis within a single automated workflow. By standardizing, computing, and writing both types of insight data back into a unified data lake warehouse, a deep integration of offensive and defensive capabilities is achieved, fundamentally enhancing the proactiveness of security defense.

[0104] In some embodiments, the self-improving, integrated offensive and defensive data lake warehouse method further includes the following steps:

[0105] In the model communication and control service, the correctness of standard structured data is verified based on a predefined data format;

[0106] If the standard structured data conforms to the predefined data format, the verification passes and the standard structured data is written to the data lake warehouse.

[0107] If the standard structured data does not conform to the predefined data format, the verification fails, the data write operation is blocked, and an error log is generated.

[0108] Specifically, the verification process is a multi-layered, automated quality control pipeline. First, syntax compliance checks are performed, verifying whether required fields of the data object are missing, whether the data type of field values ​​(such as strings, numbers, arrays) conforms to the definition, and whether specific format fields such as IP addresses match regular expressions. There are no restrictions on the type of objects being verified in this compliance check.

[0109] Secondly, semantic logic validation is performed to check whether the business logic between field values ​​is consistent. For example, a threat indicator marked as "high confidence" should have its associated "last occurrence time" within the corresponding time interval; an attack step describing "lateral movement" should have its target asset existing in a known asset list. If the above validation fails, the data write operation is intercepted, and an error log is generated for recording.

[0110] This embodiment solves the problem of writing unstructured or erroneous model output into the data lake warehouse. It prevents invalid or erroneous data from contaminating the data lake warehouse, ensuring that each data write-back enhances the overall reliability of the system.

[0111] The present embodiment will now be described and illustrated through preferred embodiments.

[0112] The self-improving, integrated offensive and defensive data lake warehouse system of this preferred embodiment is applied to scenarios of automated advanced long-term threat hunting and proactive defense.

[0113] like Figure 6As shown, the system first receives a low-confidence security alert stating: "The PowerShell process on host PC-01 exhibits suspicious network outreach behavior." This alert is automatically routed to the Model Communication and Control Service, which then passes the alert text as an initial task instruction to the pre-trained network security big data model, triggering a defensive analysis task.

[0114] The cybersecurity big data model then initiates a series of related queries to the data lake warehouse through the model communication and control service, aiming to construct a comprehensive picture of the event. The query requests are sent in natural language, specifying: "Detailed asset information for host PC-01, user login records for the past 72 hours, and the complete process tree and parent process information of the PowerShell process that triggered the alert." The bidirectional communication interface of the model communication and control service translates these requests into efficient structured query statements in real time, executes them, and returns the integrated query results to the model.

[0115] After performing correlation analysis on the returned data, the cybersecurity big data model discovered a key clue: the suspicious PowerShell process was launched by a document recently opened by a user, and the external IP address it connected to was not within any of the enterprise's legitimate business or whitelists. Based on this reasoning, the model autonomously decided to conduct proactive probing to verify the threat hypothesis. It invoked port scanning and domain reputation lookup tools built into the toolset and execution engine through the model communication and control service. Under the constraints of security policies, the model communication and control service executed these tools and fed back the probing results—"Port 443 of the target IP address is open, and the associated domain name has a suspicious reputation rating among multiple threat intelligence sources"—to the model.

[0116] Secondly, combining historical query results with new data obtained through proactive probing, the cybersecurity big data model performs comprehensive reasoning and concludes that this is highly likely a targeted phishing email attack. The target IP address is a previously undocumented Advanced Persistent Threat (APT) command and control server. Subsequently, the model generates defensive insight data. Guided by the data structuring and verification module of the model's communication and control service, the model formats this unstructured defensive insight data into a complete threat indicator object according to the STIX standard. This object is in machine-readable JSON format, and key fields include: type: ipv4-addr (threat indicator type), value: xxxx (specific value), confidence: high (confidence level), kill_chain_phases: command-and-control (attack phase), etc.

[0117] The data structuring and validation module of the model communication and control service automatically validates the generated standard structured data objects to ensure their syntactic and semantic integrity and correctness. Upon successful validation, an atomic transaction persistently writes this standard structured data as a new, high-confidence threat intelligence record into the data lake warehouse's threat intelligence table. At this point, the data lake warehouse's knowledge base is enhanced by the addition of an actionable, high-quality threat indicator, which can be immediately used by all subsequent automated detection rules or manual investigations.

[0118] Finally, after responding immediately to the current alert, the system logic automatically or the analyst triggers an offensive exploration task. The new task instruction is: "Based on the assumption that host PC-01 is considered a potentially compromised host, simulate a lateral movement path that an attacker might initiate based on this." Based on the new task objective, the network security big data model again queries the data warehouse through the model communication and control service to obtain key data related to PC-01, including the network topology map, a list of assets on the same network segment, known local or remote vulnerability information, and shared account configurations. The model utilizes its internalized attack tactics knowledge base to extrapolate attack paths and generate a detailed attack graph. This graph reveals a specific potential attack chain: an attacker might exploit an unpatched vulnerability on PC-01 to gain higher privileges, and then further leverage internal network trust relationships to move laterally to the more critical domain controller DC-02.

[0119] Guided by the model communication and control service, this attack graph was structured into a series of interconnected standard structured data objects (such as multiple Attack-Pattern objects describing specific tactics, and Relationship objects describing attack relationships between assets). After verification, this set of standard structured data describing potential threat paths was written into the attack graph table of the data lake warehouse. This marks the completion of the second self-improvement and integrated attack-defense closed loop path. The system not only processes current events but also proactively predicts and records future potential threat paths, directly transforming the results of offensive simulations into knowledge that can guide proactive defense.

[0120] Furthermore, in conjunction with the self-improving, integrated attack and defense data lake warehouse method provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the self-improving, integrated attack and defense data lake warehouse methods described in the above embodiments.

[0121] It should be noted that all information and data involved in this application are authorized by the user or fully authorized by all parties and will be used legally.

[0122] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0123] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0124] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0125] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A self-improving, integrated offensive and defensive data lake warehouse system, characterized in that, include: A data lake warehouse connects to pre-built model communication and control services to store multimodal cybersecurity data; The network security data includes standard structured data written by the model communication and control service; A pre-trained cybersecurity big data model is connected to the model communication and control service to initiate data query requests to the model communication and control service, so that the model communication and control service performs data query operations on the data lake warehouse according to the data query requests and returns the query results; The query results returned by the model communication and control service are analyzed and reasoned to generate insight data; The model communication and control service generates the standard structured data based on the insight data and writes the standard structured data into the data lake warehouse.

2. The self-improving, integrated offensive and defensive data lake warehouse system according to claim 1, characterized in that, The model communication and control service includes a two-way communication interface; The bidirectional communication interface is connected to the data lake warehouse and the network security big model respectively, and is used to convert the data query request of the network security big model into a structured query of the data lake warehouse, and return the query result to the network security big model in natural language or structured format.

3. The self-improving, integrated offensive and defensive data lake warehouse system according to claim 1, characterized in that, The model communication and control service includes a toolset and an execution engine; The toolset and execution engine are connected to the cybersecurity big model and are used to provide one or more executable security operation tools for the cybersecurity big model.

4. The self-improving, integrated offensive and defensive data lake warehouse system according to claim 3, characterized in that, The security operation tools include port scanning tools, asset information query tools, domain name reputation query tools, or attack graph generation tools; The port scanning tool is connected to the network security big model and is used to probe specified network addresses and ports; The asset information query tool is connected to the network security big model and is used to retrieve the corresponding hardware configuration information based on the asset identifier. The domain name reputation query tool is connected to the network security big data model and is used to query the reputation assessment results based on a specified domain name. The attack graph generation tool is connected to the network security big model and is used to construct a structured attack graph describing the attack steps.

5. The self-improving, integrated offensive and defensive data lake warehouse system according to claim 1, characterized in that, The model communication and control service includes a data structuring and verification module; The data structuring and verification module is connected to the network security big model and is used to convert the insight data generated by the network security big model into standard structured data that conforms to a predefined data pattern.

6. The self-improving, integrated offensive and defensive data lake warehouse system according to claim 1, characterized in that, The insight data includes defensive insight data and offensive insight data; The defensive insight data includes threat indicator types, threat indicator values, confidence levels, and attack stages; The attack insight data includes a structured attack graph that describes the attack steps.

7. A self-improving, integrated offensive and defensive data lake warehouse method, characterized in that, The method is applicable to the self-improving, integrated attack and defense data lake warehouse system as described in any one of claims 1 to 6, and the method includes: The pre-trained cybersecurity big data model initiates a data query request to the pre-built model communication and control service; In the model communication and control service, a data query operation is performed on the data lake warehouse according to the data query request, and the query result is returned to the network security big model; Within the aforementioned cybersecurity big data model, the query results are analyzed and reasoned to generate insightful data. In the model communication and control service, standard structured data is generated based on the insight data, and the standard structured data is written into the data lake warehouse.

8. The self-improving, integrated offensive and defensive data lake warehouse method according to claim 7, characterized in that, The cybersecurity big data model analyzes and reasons based on the query results to generate insight data, and also includes: If it is the defensive analysis phase, the network security big model identifies potential security threats based on the query results and generates defensive insight data that includes threat indicator types, threat indicator values, confidence levels, and attack phases. If it is the attack analysis phase, the network security big model simulates attacker behavior based on the query results and generates attack insight data containing a structured attack graph.

9. The self-improving, integrated offensive and defensive data lake warehouse method according to claim 7, characterized in that, The process of generating standard structured data and writing the standard structured data into the data lake warehouse includes: In the model communication and control service, the correctness of the standard structured data is verified based on a predefined data format; If the standard structured data conforms to the predefined data format, the verification is successful, and the standard structured data is written into the data lake warehouse. If the standard structured data does not conform to the predefined data format, the verification fails, the data write operation is blocked, and an error log is generated.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the self-improving integrated offensive and defensive data lake warehouse method as described in any one of claims 7 to 9.

Images