Cloud-native AI detection adaptive method and system

Abstract

The present application relates to the technical field of cloud native, and provides an AI detection adaptive method and system based on cloud native, wherein the method comprises the following steps: obtaining original log data, and distributing the obtained original log data to different queues according to different protocols; using different model detection containers to detect data obtained from the corresponding queues, and sending the detection results to corresponding alarm channels; monitoring the detection situation of data in each queue, and formulating a control strategy to adjust the running model detection container according to the detection situation. The AI detection adaptive method and system based on cloud native according to the exemplary embodiments of the present application can realize fine isolation and dynamic adjustment of resources and horizontal scalability of models.

Description

Technical Field

[0001] This invention relates to the field of cloud-native technology, and in particular to a cloud-native AI detection adaptive method and system. Background Technology

[0002] In the cybersecurity field, most products are delivered and sold as a single physical server coupled with software deployment. In this scenario, most AI detection models used for threat detection are deployed directly on physical servers, achieving real-time detection by running multiple processes or threads.

[0003] In practical applications, the existing model has the following shortcomings: 1. It cannot achieve precise resource isolation. Multi-process scheduling cannot accurately allocate and schedule CPU and memory, and threads cannot be isolated by allocating resources. 2. In actual operation, it is necessary to analyze the amount of data on the live network in advance to pre-allocate the number of different models. This model is relatively rigid. If too many resources are allocated, it will lead to resource waste. If too few resources are allocated, performance bottlenecks will occur during operation. Moreover, if the amount of data accessed is uneven, with peaks and troughs, problems will arise no matter how the allocation is done. 3. When multiple models are running under a framework, the number of different models launched is proportionally allocated. However, different AI detection modes use different algorithms, resulting in different detection speeds. This makes it impossible to achieve horizontal scaling based on the different detection speeds of different detection models.

[0004] Therefore, how to provide an AI model detection method that can achieve precise resource isolation and allocation and support the horizontal expansion of the detection model has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of this, in order to solve the problems of inaccurate resource isolation and allocation and the inability of detection models to scale horizontally, this invention provides a cloud-native AI detection adaptive method and system by achieving resource isolation and detection model expansion through a cloud-native architecture, and by achieving accurate resource allocation and adjustment through monitoring and core control.

[0006] On the one hand, this invention provides a cloud-native AI detection adaptive method, including:

[0007] Step S1: Obtain raw log data and distribute the obtained raw log data to different queues according to different protocols;

[0008] Step S2: Use different model detection containers to obtain data from the corresponding queues for detection, and send the detection results to the corresponding alarm channels;

[0009] Step S3: Monitor the detection status of data in each queue, and formulate control strategies to adjust the running model detection container based on the detection status.

[0010] Furthermore, in step S1 of the cloud-native AI detection adaptive method of the present invention, the raw log data is the raw log data of a layer 4 or layer 7 protocol.

[0011] Furthermore, in step S2 of the cloud-native AI detection adaptive method of the present invention, different model detection containers are used to obtain data from the corresponding queues for detection, including: creating a container detection group for model detection containers of the same detection mode, and obtaining data from the corresponding queues for detection through the model detection containers of the container detection group.

[0012] Furthermore, step S3 of the cloud-native AI detection adaptive method of the present invention includes: monitoring the original data import speed of each queue and the data detection speed of the container detection group corresponding to the queue, formulating a control strategy based on the change in the difference between the original data import speed and the data detection speed, and adjusting the number of model detection containers according to the control strategy.

[0013] Furthermore, step S3 of the cloud-native AI detection adaptive method of the present invention includes:

[0014] When the difference between the original data import speed and the data detection speed is greater than zero, increase the number of model detection containers until the original data import speed and the data detection speed are equal.

[0015] When the difference between the original data import speed and the data detection speed is less than zero, reduce the number of model detection containers until the original data import speed and the data detection speed are equal.

[0016] When the difference between the original data import speed and the data detection speed is equal to zero, the number of model detection containers remains unchanged.

[0017] On the other hand, the present invention provides a cloud-native AI detection adaptive system, comprising:

[0018] The data import module is used to acquire raw log data and distribute the acquired raw log data to different queues according to different protocols.

[0019] The queue information module is used to store the raw log data imported by the import module into the corresponding queue according to different protocols;

[0020] The K8s container module is used to run different model detection containers, which are used to retrieve data from corresponding queues for detection.

[0021] The core controller is used to monitor the detection status of data in each queue and formulate control strategies based on the detection status.

[0022] The K8s control module is used to adjust the running model detection container according to the control strategy defined by the core controller.

[0023] The alarm data module has multiple alarm channels for receiving detection results sent by each model detection container.

[0024] Furthermore, this invention is based on a cloud-native AI detection adaptive system. The model detection container in the K8s container module is used to obtain data from the corresponding queue for detection and send the detection results to the corresponding alarm channel.

[0025] Furthermore, in the K8s container module of the cloud-native AI detection adaptive system of this invention, model detection containers of the same detection mode belong to the same container detection group.

[0026] Furthermore, the core controller of the cloud-native AI detection adaptive system of the present invention is specifically used to: monitor the original data import speed of each queue and the data detection speed of the container detection group corresponding to the queue, and formulate a control strategy based on the change in the difference between the original data import speed and the data detection speed.

[0027] Furthermore, the K8s control module of the cloud-native AI detection adaptive system of this invention is specifically used for:

[0028] When the difference between the original data import speed and the data detection speed is greater than zero, increase the number of model detection containers until the original data import speed and the data detection speed are equal.

[0029] When the difference between the original data import speed and the data detection speed is less than zero, reduce the number of model detection containers until the original data import speed and the data detection speed are equal.

[0030] When the difference between the original data import speed and the data detection speed is equal to zero, the number of model detection containers remains unchanged.

[0031] This invention, based on a cloud-native AI detection adaptive method and system, has the following beneficial effects:

[0032] 1. The cloud-native underlying architecture can effectively isolate resources. All detection models are not deployed directly on physical servers, but on a Kubernetes-based underlying architecture, achieving fine-grained resource isolation.

[0033] 2. By importing and monitoring real-time data, dynamic resource adjustments for different models are achieved without manual intervention, thus avoiding performance bottlenecks and resource waste.

[0034] 3. Based on the cloud-native Kubernetes architecture, instead of using a large detection framework to include the operation of various models, different models can be decomposed, and each model can run independently according to its own characteristics. A unified data output interface can be used to achieve independent control of the model. Resources can be allocated according to actual needs to maximize resource utilization and the horizontal scalability of the model. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of a cloud-native AI detection adaptive method according to an exemplary first embodiment of the present invention.

[0037] Figure 2 This is an architecture diagram of a cloud-native AI detection adaptive system, which is an exemplary third embodiment of the present invention.

[0038] Figure 3 This is an exemplary fifth embodiment of the present invention, showing the application principle of a cloud-native AI detection adaptive system. Detailed Implementation

[0039] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0040] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0041] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0042] The theoretical technology of this invention is as follows:

[0043] The cloud-native underlying architecture can effectively isolate resources. All detection models are not deployed directly on physical servers, but on a Kubernetes-based underlying architecture, achieving fine-grained resource isolation.

[0044] The core controller module monitors the consumption progress of queue consumers and analyzes whether the current consumption speed is lagging. If consumption is lagging, it means that the current number of replicas of the detection model is insufficient to support the current detection speed of the raw logs. In this case, the core controller module will call the Kubernetes API to increase the number of replicas for the application load of the detection model, observe, and repeat the above process until the detection process reaches a dynamic balance. If the data volume decreases, then it is unnecessary to maintain that many replicas, otherwise it would be a waste of resources. In this case, the core controller will call the Kubernetes API to reduce the number of replicas for the application load of the detection model. It is important to note that each model must ensure that at least one instance is running, even if no data is coming in, so that it can continue to run and be processed promptly when data arrives.

[0045] Because of the use of cloud-native Kubernetes architecture, there is no need to use a large detection framework to include the operation of each model. Different models can be split up, and each model can run independently according to its own characteristics, using a unified data output interface. In this way, different models can be controlled independently, maximizing resource utilization.

[0046] The following definitions of terms used in the embodiments are as follows:

[0047] Kubernetes (k8s), also known as k8s, is an open-source platform that automates the operation of Linux containers. It helps users avoid many manual deployment and scaling operations in the application containerization process. Multiple groups of hosts running Linux containers can be aggregated together, and Kubernetes helps you easily and efficiently manage these clusters. Moreover, these clusters can be deployed across public, private, or hybrid clouds.

[0048] DNS: Domain Name System (DNS) is a service on the Internet. As a distributed database that maps domain names to IP addresses, it makes it easier for people to access the Internet. DNS uses TCP and UDP port 53. The DNS protocol is used to translate domain names into IP addresses (and vice versa).

[0049] DGA, short for Domain Generate Algorithm, is a pseudo-random domain name generation algorithm that can generate a large number of pseudo-random domain names in batches. These domain names are used as C&C (command and control server) domains. Because the algorithm and parameters can be predetermined, the server can use the same algorithm to generate the same domain names as the client. When an attacker needs to control a victim's machine, they can register one or more of these domain names. The victim's machine, acting as a client, will periodically attempt to connect to the C&C server. Once the connection is successful, the malicious program can receive instructions from the server and perform a series of malicious activities such as information gathering and malicious damage.

[0050] HTTP, short for Hyper Text Transfer Protocol, is a protocol used to transfer hypertext from World Wide Web (WWW) servers to local browsers. HTTP is based on the TCP / IP communication protocol for transmitting data (HTML files, image files, query results, etc.). The HTTP protocol operates on a client-server architecture. The browser, acting as an HTTP client, sends all requests to the HTTP server, i.e., the web server, via a URL. The web server, upon receiving the request, sends a response to the client.

[0051] SSH (Secure Shell) is an encrypted network protocol that provides a secure environment for network services in insecure networks. SSH establishes a secure tunnel between the SSH client and server. The most common use of SSH is for remote system login; it's frequently used to transmit command-line interfaces and execute commands remotely. SSH is most frequently used on Unix-like systems, but it can also be used to a limited extent on Windows operating systems. Windows 10 version 1803 provides the OpenSSH tool.

[0052] Syslog is a data protocol used by various devices on a network to collect logs to a log server. It is supported by almost all network devices and can record log messages of various event types. Common devices that support syslog include routers, switches, printers, and even Unix-like servers can generate syslog messages to record user logins, firewall events, Apache or Nginx access logs, etc.

[0053] Figure 1 Here is a flowchart of a cloud-native AI detection adaptive method according to an exemplary first embodiment of the present invention, as shown below. Figure 1 As shown, the method in this embodiment includes:

[0054] Step S1: Obtain raw log data and distribute the obtained raw log data to different queues according to different protocols;

[0055] Step S2: Use different model detection containers to obtain data from the corresponding queues for detection, and send the detection results to the corresponding alarm channels;

[0056] Step S3: Monitor the detection status of data in each queue, and formulate control strategies to adjust the running model detection container based on the detection status.

[0057] The raw log data in step S1 of this embodiment is raw log data of layer 4 or layer 7 protocols, such as HTTP, DNS, FTP, SSH, TCP, UDP, etc. In practical applications, raw data can be imported using methods such as syslog.

[0058] In step S2 of this embodiment, different model detection containers are used to obtain data from corresponding queues for detection. This includes: creating a container detection group for model detection containers of the same detection mode, and obtaining data from the corresponding queues through the model detection containers of the container detection group for detection. In this way, the container detection group will not detect data repeatedly, realizing the horizontal expansion of the model detection containers.

[0059] In this embodiment, in order to enable multiple model detection containers to consume data from the same queue, the number of partitions in the queue is not less than the number of model detection containers in the container detection group; otherwise, the extra model detection containers will be idle.

[0060] Step S3 of the method in this embodiment includes: monitoring the original data import speed of each queue and the data detection speed of the container detection group corresponding to the queue; formulating a control strategy based on the change in the difference between the original data import speed and the data detection speed; and adjusting the number of model detection containers according to the control strategy.

[0061] The second exemplary embodiment of the present invention provides a cloud-native AI detection adaptive method. This embodiment is... Figure 1 In a preferred embodiment of the method shown, step S3 of the method in this embodiment includes:

[0062] When the difference between the original data import speed and the data detection speed is greater than zero, increase the number of model detection containers until the original data import speed and the data detection speed are equal.

[0063] When the difference between the original data import speed and the data detection speed is less than zero, reduce the number of model detection containers until the original data import speed and the data detection speed are equal.

[0064] When the difference between the original data import speed and the data detection speed is equal to zero, the number of model detection containers remains unchanged.

[0065] In this embodiment, when data backlog occurs or the model detection container is idle, the number of model detection containers can be adjusted until data import and data detection reach a balance.

[0066] Figure 2 To provide a cloud-native AI detection adaptive system according to an exemplary third embodiment of the present invention, the system of this embodiment includes:

[0067] The data import module is used to acquire raw log data and distribute the acquired raw log data to different queues according to different protocols.

[0068] The queue information module is used to store the raw log data imported by the import module into the corresponding queue according to different protocols;

[0069] The K8s container module is used to run different model detection containers, which are used to retrieve data from corresponding queues for detection.

[0070] The core controller is used to monitor the detection status of data in each queue and formulate control strategies based on the detection status.

[0071] The K8s control module is used to adjust the running model detection container according to the control strategy defined by the core controller.

[0072] The alarm data module has multiple alarm channels for receiving detection results sent by each model detection container.

[0073] In this cloud-native AI detection adaptive system, the model detection container in the K8s container module is used to obtain data from the corresponding queue for detection and send the detection results to the corresponding alarm channel for display by upper-layer services.

[0074] In the Kubernetes container module of the cloud-native AI detection adaptive system in this embodiment, model detection containers of the same detection mode belong to the same container detection group. Each type of detection model acts as a stateless deployed application load, and the number of model detection containers started during actual operation is dynamically scheduled by the core controller.

[0075] In this embodiment of the cloud-native AI detection adaptive system, the core controller is specifically used to: monitor the raw data import speed of each queue and the data detection speed of the container detection group corresponding to the queue, and formulate a control strategy based on the change in the difference between the raw data import speed and the data detection speed.

[0076] The fourth exemplary embodiment of the present invention provides a cloud-native AI detection adaptive system. This embodiment is... Figure 2 In a preferred embodiment of the system shown, the K8s control module of this embodiment is specifically used for:

[0077] When the difference between the original data import speed and the data detection speed is greater than zero, increase the number of model detection containers until the original data import speed and the data detection speed are equal.

[0078] When the difference between the original data import speed and the data detection speed is less than zero, reduce the number of model detection containers until the original data import speed and the data detection speed are equal.

[0079] When the difference between the original data import speed and the data detection speed is equal to zero, the number of model detection containers remains unchanged.

[0080] Figure 3This embodiment is a cloud-native AI detection adaptive system according to an exemplary fifth embodiment of the present invention. Figure 2 The preferred embodiment of the system shown describes the application principle of the system as follows:

[0081] The system acquires raw log data and distributes it to DNS protocol raw log queues, HTTP protocol raw log queues, and SSH protocol raw log queues according to different protocols. A DGA model detection container retrieves data from the DNS protocol raw log queue for detection and sends the results to the DGA alarm channel. Similarly, an HTTP covert tunnel model detection container retrieves data from the HTTP protocol raw log queue for detection and sends the results to the HTTP covert tunnel alarm channel. Finally, an SSH brute-force attack model detection container retrieves data from the SSH protocol raw log queue for detection and sends the results to the SSH brute-force attack alarm channel.

[0082] The core controller monitors the raw data import speed of each queue and the data detection speed of the container detection group corresponding to that queue. A control strategy is formulated based on the change in the difference between the raw data import speed and the data detection speed. A Kubernetes control module adjusts the number of model detection containers according to the control strategy. When the difference between the raw data import speed and the data detection speed is greater than zero, the number of model detection containers is increased until the raw data import speed and the data detection speed are equal. When the difference between the raw data import speed and the data detection speed is less than zero, the number of model detection containers is decreased until the raw data import speed and the data detection speed are equal. When the difference between the raw data import speed and the data detection speed is equal to zero, the number of model detection containers remains unchanged.

[0083] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cloud-native AI detection adaptive method, characterized in that, The method comprises: Step S1: obtaining original log data, and distributing the obtained original log data to different queues according to different protocols; Step S2: acquiring data from the corresponding queue by using different model detection containers for detection, and sending the detection result to the corresponding alarm channel; Step S3: monitoring the detection situation of the data in each queue, and formulating a control strategy to adjust the running model detection container according to the detection situation; Step S3 comprises: monitoring the original data import speed of each queue and the data detection speed of the container detection group corresponding to the queue, formulating a control strategy according to the difference between the original data import speed and the data detection speed, and adjusting the number of model detection containers according to the control strategy; Step S3 comprises: When the difference between the original data import speed and the data detection speed is greater than zero, the number of model detection containers is increased until the original data import speed and the data detection speed are equal; When the difference between the original data import speed and the data detection speed is less than zero, the number of model detection containers is reduced until the original data import speed and the data detection speed are equal; When the difference between the original data import speed and the data detection speed is equal to zero, the number of model detection containers is kept unchanged. 2.The cloud-native based AI detection adaptive method according to claim 1, wherein, The original log data in step S1 is four-layer or seven-layer protocol original log data. 3.The cloud-native based AI detection adaptation method of claim 1, wherein, In step S2, the different model detection containers acquire data from the corresponding queue for detection, which comprises: creating a container detection group for model detection containers of the same detection mode, and acquiring data from the corresponding queue by the model detection containers of the container detection group for detection.

4. An AI detection adaptive system based on cloud origin, the system comprising: a data import module for obtaining original log data, and distributing the obtained original log data to different queues according to different protocols; a queue information module for storing the original log data imported by the import module to the corresponding queue according to different protocols; a K8s container module for running different model detection containers, the model detection containers being used to acquire data from the corresponding queue for detection; a core controller for monitoring the detection situation of the data in each queue, and formulating a control strategy according to the detection situation; a K8s control module for adjusting the running model detection container according to the control strategy formulated by the core controller; an alarm data module having a plurality of alarm channels, and being used to receive the detection result sent by each model detection container; the core controller is specifically used for: monitoring the original data import speed of each queue and the data detection speed of the container detection group corresponding to the queue, and formulating a control strategy according to the difference between the original data import speed and the data detection speed; the K8s control module is specifically used for: when the difference between the original data import speed and the data detection speed is greater than zero, the number of model detection containers is increased until the original data import speed and the data detection speed are equal; when the difference between the original data import speed and the data detection speed is less than zero, the number of model detection containers is reduced until the original data import speed and the data detection speed are equal; When the difference between the original data import speed and the data detection speed is equal to zero, the number of model detection containers is kept unchanged. 5.The cloud-native based AI detection adaptive system of claim 4, wherein, The model detection container in the K8s container module is used to obtain data from the corresponding queue for detection, and send the detection result to the corresponding alarm channel. 6.The cloud-native based AI detection adaptive system of claim 4, wherein, In the K8s container module, the model detection containers of the same detection mode belong to the same container detection group.

Images