Fine-grained Traceability Data Acquisition Method and System for Cloud Platform Based on Multi-layer Data Fusion

Abstract

The present invention discloses a method and system for collecting fine-grained traceability data of a cloud platform based on multi-layer data fusion, belonging to the field of cloud native technologies. The present invention uses the eBPF hook method to collect original function call information from each abstraction layer of the cloud platform; for different abstraction layers in the cloud platform, eBPF stores the function call information in different key mapping tables according to the characteristics of the function calls, and the key mapping tables include a TCP mapping table, a container orchestration mapping table, a container mapping table, and a file mapping table; optimize the trace information of the function call information of different abstraction layers; associate the optimized information of different abstraction layers according to the keys and values in different mapping tables to obtain a cross-layer information set; generate a cross-layer traceability graph according to the cross-layer information set. The present invention can solve the problems of unclear entity, function, and container mapping relationships, unclear ownership, and difficulty in realizing efficient and unified cross-layer attack traceability in the cloud platform.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud native, and particularly relates to a method and system for collecting fine-grained traceability data of a cloud platform based on multi-layer data fusion. Background Art

[0002] In recent years, cloud native technology has developed rapidly. Through technologies such as virtualization, container cluster management, microservices, and serverless computing, efficient resource allocation across operating systems and physical machines has been achieved, and a function sharing system and deployment system across applications have been constructed. These technologies support the rapid construction of modern application environments and bring great value in terms of innovation speed, cost optimization, etc.

[0003] To enable fine-grained regulation and utilization of resources, cloud native platforms have multiple abstraction layers, including: physical layer, operating system layer, container orchestration layer, and virtualization layer, etc. Each layer virtualizes resources. However, virtualization technology also breaks the security protection boundaries of traditional hosts and applications, expands the semantic gap between upper-layer application behaviors and lower-layer system behaviors, resulting in unclear attribution of entities and behaviors and unclear mapping relationships in cloud native platforms. Therefore, when performing global attack detection and traceability analysis on cloud platforms, it is necessary to collect cross-layer and multi-source data and perform unified analysis.

[0004] Existing data collection solutions cannot perceive resources after multi-layer virtualization such as cloud native applications, container orchestration, and container runtimes, cannot accurately divide the attribution of behaviors, and will ignore mapping relationships such as files and networks. The loss of this information is likely to cause key attack paths to be missed, ultimately resulting in the failure of attack detection. Summary of the Invention

[0005] To solve the problems in the prior art, the present invention proposes a method and system for collecting fine-grained traceability data of a cloud platform based on multi-layer data fusion to solve the problems of unclear mapping relationships, unclear attribution of entities, functions, and containers in cloud platforms, and difficulty in achieving efficient and unified cross-layer attack traceability.

[0006] In a first aspect, an embodiment of the present invention provides a method for collecting fine-grained traceability data of a cloud platform based on multi-layer data fusion, including: collecting original function call information from each abstraction layer of the cloud platform by using the eBPF hook method; for different abstraction layers in the cloud platform, eBPF stores the function call information in different key mapping tables according to the characteristics of the function calls, and the key mapping tables include a TCP mapping table, a container orchestration mapping table, a container mapping table, and a file mapping table; optimizing the traceability information of the function call information of different abstraction layers; associating the optimized information of different abstraction layers according to the keys and values in different mapping tables to obtain a cross-layer information set; and generating a cross-layer traceability graph according to the cross-layer information set.

[0007] Second aspect, an embodiment of the present invention provides a fine-grained traceability data acquisition system for a cloud platform based on multi-layer data fusion, including: an information collection module that collects original function call information from each abstraction layer of the cloud platform using the eBPF hook method; an eBPF mapping table storage module that, for different abstraction layers in the cloud platform, stores function call information in different key mapping tables according to the characteristics of function calls by eBPF; a trace information optimization module that optimizes trace information for function call information of different abstraction layers; a cross-layer information association module that associates information of different abstraction layers optimized by the trace information optimization module according to keys and values in different mapping tables to obtain a cross-layer information set; a cross-layer traceability graph generation module that generates a cross-layer traceability graph according to the cross-layer information set of the cross-layer information association module.

[0008] Third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the fine-grained traceability data acquisition method for a cloud platform based on multi-layer data fusion as described above.

[0009] Fourth aspect, an embodiment of the present invention provides a readable storage medium, in which a computer program is stored, and the computer program includes program code for controlling a process to execute the process, and the process includes the fine-grained traceability data acquisition method for a cloud platform based on multi-layer data fusion as described above.

[0010] Compared with the prior art, the present invention collects key entity and behavior data from each layer, constructs key mapping tables for virtualized resources such as files, processes, and networks, realizes the correct attribution of system behaviors and the correct parsing and fusion of entities in each layer, and finally constructs a unified data model to uniformly represent the behaviors of system entities, providing an accurate data source for security requirements such as process monitoring, behavior auditing, and threat traceability in cloud native platforms, solving the problem of difficult to efficiently construct global observability in cloud platforms, and solving the problems of unclear mapping relationships, unclear attribution, and difficult to achieve efficient unified cross-layer attack traceability of entities, functions, and containers in cloud platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a framework diagram of the fine-grained traceability data acquisition method for a cloud platform based on multi-layer data fusion according to an embodiment of the present invention;

[0012] Figure 2 is a flow chart of trace information optimization according to an embodiment of the present invention;

[0013] Figure 3 is a flow chart of cross-layer information association according to an embodiment of the present invention;

[0014] Figure 4 This is the flowchart for generating the cross-layer traceability graph of an embodiment of the present invention. Specific embodiments

[0015] The following further elaborates and explains the present invention in conjunction with specific embodiments. The described embodiments are merely demonstrations of the disclosed content and do not delimit the scope of limitation. Without conflict, the technical features of each embodiment of the present invention can be combined accordingly.

[0016] Before introducing the present invention, first, the keywords for understanding the present invention are explained:

[0017] Key-based eBPF Map: A key-based mapping is a data structure used to store and retrieve key-value pairs. Each value is associated with a unique key, so data can be accessed and retrieved through a specific key. In a cloud platform, after multiple virtualizations, the function ID and container ID can uniquely determine a function instance and a container instance.

[0018] The abstraction layer of the cloud-native platform: In the cloud-native platform, each abstraction layer provides different functions and services to support the development, deployment, and operation of applications. The abstraction layer mainly includes: the physical layer, the virtualization layer, and the container orchestration layer. Among them, the physical layer is the foundation of the entire cloud computing infrastructure, which mainly includes hardware resources such as physical servers, storage devices, and network devices. The virtualization layer abstracts physical hardware resources into multiple virtual resources through virtualization technology, and the container orchestration layer is used to manage and coordinate the deployment, operation, and expansion of containers.

[0019] Function / function call: In the cloud-native platform, a function is an independent and executable code unit, usually used to complete specific tasks or handle specific events. Functions can be written in multiple programming languages. A function call refers to the process of triggering and executing a function. In the cloud-native platform, function calls are usually triggered by specific events, such as HTTP requests, database changes, message queue events, scheduled tasks, etc.

[0020] Container: In the cloud-native environment, a container is a lightweight and portable virtualization technology used to package and isolate applications and their dependencies. Containers provide a consistent running environment, enabling applications to run seamlessly in different computing environments.

[0021] The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

[0022] The overall system design architecture of the invention is as Figure 1As shown in the figure, the original data acquisition system is based on eBPF (Extended Berkeley Packet Filter). eBPF is a Linux kernel component that can perform dynamic programming on the kernel (most cloud-native platforms are built on Linux), and can stably collect data from specified positions in the kernel and perform simple processing. In the present invention, eBPF hooks are attached at different layers of the cloud-native system to collect original data, and the events and entities in the cloud platform are correctly attributed and mapped through a key mapping table. Among them, the construction of the key mapping table requires the management data of each abstract layer of the cloud-native technology stack. Finally, this project will perform cross-layer information association for multiple abstract layers of the cloud-native platform, parse and fuse this information into a unified data format, and generate a cross-layer traceability graph.

[0023] As Figure 1 shown, in a specific embodiment of the present invention, the method for collecting fine-grained traceability data of a cloud platform based on multi-layer data fusion includes the following steps:

[0024] 1) Use the eBPF hook method to collect original function call information from each abstract layer of the cloud platform;

[0025] 2) For different abstract layers in the cloud platform, eBPF stores the function call information in different key mapping tables according to the characteristics of the function calls,

[0026] 3) Optimize the traceability information of the function call information of different abstract layers;

[0027] 4) According to the keys and values in different mapping tables, associate the information of different abstract layers optimized in step 3) to obtain a cross-layer information set;

[0028] 5) Generate a cross-layer traceability graph according to the cross-layer information set.

[0029] It should be noted that the abstract layers of the cloud platform include the physical layer, the operating system layer, the container orchestration layer, the virtualization layer, etc. In the above method steps, step 1) works in the kernel space; step 3) works in the user space. Therefore, in step 3), data needs to be sent from the kernel space to the user space.

[0030] In a specific embodiment of the present invention, the present invention attaches eBPF hooks of different layers to different functions in a unified manner. eBPF is an instruction set and execution environment in the Linux kernel. It allows custom but constrained functions to be securely executed at different positions inside the kernel. Regardless of the technology used, hooks can be attached at different abstract layers.

[0031] In a specific embodiment of the present invention, the key mapping table correctly attributes and maps the events in the data to entities. According to the storage type, there are a total of 4 types of key storage mapping tables, namely the TCP mapping table, the container orchestration mapping table, the container mapping table, and the file mapping table; the mapping table includes a key field and a value field, the key is unique, and a value can be uniquely determined through the key.

[0032] Among them, the TCP mapping table is used to store socket information, and the key field includes:

[0033] 1) Source IP address;

[0034] 2) Port number.

[0035] The value field includes:

[0036] 1) Destination IP address port;

[0037] 2) Port number.

[0038] The container orchestration mapping table is used to record functions and corresponding container information, and the key field includes:

[0039] 1) Function;

[0040] 2) Instance where the function is located.

[0041] The value field includes:

[0042] 1) Container ID;

[0043] 2) IP address.

[0044] The container mapping table is used to record container information, and the key field includes:

[0045] 1) Container ID.

[0046] The value field includes:

[0047] 1) Real file name corresponding to the container;

[0048] 2) Process ID pid;

[0049] 3) Specific operation to be executed;

[0050] 4) Timestamp.

[0051] The file mapping table records the mounted file information, and the key field includes:

[0052] 1) Container ID;

[0053] 2) File name.

[0054] The value field includes:

[0055] 1) Mounted file name.

[0056] Exemplarily, this embodiment gives typical examples of the TCP mapping table, the container orchestration mapping table, the container mapping table, and the file mapping table, as shown in Tables 1 - 4 respectively. It should be noted that these tables are all extensible.

[0057] Table 1 TCP Key - Value Mapping Table (Extensible)

[0058]

[0059] Table 2 Container Orchestration Key - Value Mapping Table (Extensible)

[0060]

[0061] Table 3 Container Key - Value Mapping Table (Extensible)

[0062]

[0063] Table 4 File Mapping Table (Extensible)

[0064]

[0065] Typical but not limiting, the method for obtaining the key mapping tables is as follows: The additional eBPF hooks collect metrics according to the characteristics of function calls and store them in different mappings. For the TCP mapping table, the hooks record function calls through the API gateway. When the user calls the function at the container orchestration layer, the TCP mapping table records the TCP connections when the user calls the function at the container orchestration layer. For the container orchestration mapping table, the call will be scheduled by the controller in the presentation layer to an instance of the function. The orchestration information can be extracted from the controller (such as kube - apiserver) and stored in the configuration map. For the container mapping table, the hooks record a series of events generated by the function instances called in the container (these events occur in the container at the virtualization layer) and add them to the mapping table. However, some files in the container are mounted from the host. Operations on the files in the container actually involve the host at the operating system layer. Therefore, for the file mapping table, the hooks record the real paths of the files in the function instances in the file mapping table. By attaching hooks at different layers and storing cross - layer data in different mapping tables, cross - layer tracking is achieved.

[0066] To achieve non - intrusive and high - performance tracing, the present invention adopts a host - based deployment method and proposes a tracing information optimization algorithm. The tracing module is deployed as a container on each worker node without the need to additionally deploy components inside the function instances. The cloud service provider does not need to make any modifications to the user functions.

[0067] As described above, the information collection module works in the kernel space, while the optimization process is in the user space. The design goal of trace information optimization is to reduce a large number of duplicate data records and reduce the system overhead of generating the traceability graph.

[0068] As Figure 2 shown, the optimization process is as follows:

[0069] 3.1) Send data from the kernel space to the user space;

[0070] 3.2) Construct a set V for recording container IDs and a container data set D for collecting data from containers, both of which are set to empty sets, and process each row of the collected data from the key mapping table; if the container ID of this data is already in the set V, it means that the sent data has already recorded events from this container, and at this time, directly extract the container data set D corresponding to this container; if the container ID of this data is not in the set V, then skip this data;

[0071] 3.3) Determine whether this data already exists in the container data set D corresponding to this container. If the data already exists, then skip this data. If this data does not exist, then add this data to the container data set D;

[0072] 3.4) Calculate the difference between the timestamp of this data and the timestamp of the earliest data stored in the container data set D; if the difference exceeds one minute, the data of this container will be compressed using the gzip method to obtain the optimized trace information table.

[0073] The information collection work is carried out at different layers in the cloud-native system, including the physical layer, the operating system layer, and the virtualization layer, etc. Specifically, as Figure 3 shown, the information of different abstraction layers is associated, which includes the following sub-steps:

[0074] 4.1) Take each item in the optimized trace information table as a single data record for input, and judge whether there is a corresponding data record in the TCP mapping table according to the destination IP address in the TCP mapping table. If not, skip the subsequent steps and directly process the next input data record; if there is such a data record, then enter step 4.2);

[0075] 4.2) Judge whether the destination IP address is the same as the IP address in the container orchestration mapping table. If they are the same, then add the function instance in the container orchestration mapping table to the cross-layer information set. If they are different, then skip the subsequent steps and return to step 4.1);

[0076] 4.3) Determine whether the container IDs in the container mapping table and the container orchestration mapping table are the same, and determine whether the container IDs in the file mapping table and the container orchestration mapping table are the same; if any of them is different, do not store this data record; if both are the same, add the function information and IP address in the container orchestration mapping table to the cross-layer information set for storage; the function information in the cross-layer information set includes the real file name, process ID pid, specific operation performed, timestamp, and file name after mounting.

[0077] 4.4) Determine whether the function instance has terminated. If it has terminated, proceed to step 5) to generate the cross-layer traceability graph; otherwise, return to step 4.1) to process the next input data record.

[0078] The purpose of generating the cross-layer traceability graph is to facilitate security analysts in constructing a global observation view of the cloud platform to perform attack detection tasks. To achieve efficient detection and system analysis, a complete traceability graph is constructed only when each function instance dies. This strategy can ensure the timeliness of generating the traceability graph, improve the detection efficiency, and reduce the analysis cost.

[0079] As Figure 4 shown, in a specific embodiment of the present invention, generating the cross-layer traceability graph according to the cross-layer information set includes the following sub-steps:

[0080] 5.1) Input the container ID in the cross-layer information set. If it is equal to the container ID for which the traceability graph is to be generated, add the data corresponding to the container ID to the traceability graph data set; the traceability graph data set includes the container traceability graph, function instance traceability graph, function traceability graph, and global traceability graph.

[0081] 5.2) Repeat step 5.1) until all data is processed.

[0082] 5.3) Add the function information in the cross-layer information set as nodes in the container traceability graph, and use the function call information as the edges connecting the nodes to generate the container traceability graph.

[0083] 5.4) First, add the container traceability graph to the corresponding function instance traceability graph, and then add the function instance traceability graph to the corresponding function traceability graph; add the function traceability graph to the global traceability graph to obtain the cross-layer traceability graph.

[0084] As can be seen from the above introduction, the present invention collects key entities and behavior data from each layer, constructs key mapping tables for virtualized resources such as files, processes, and networks, realizes the correct attribution of system behaviors and the correct parsing and fusion of entities in each layer, and finally constructs a unified data model to uniformly represent the behaviors of system entities, providing an accurate data source for security requirements such as process monitoring, behavior auditing, and threat traceability in the cloud native platform, and solving the problem of difficult to efficiently build global observability in the cloud platform.

[0085] Based on the same concept, an embodiment of the present invention provides a fine-grained traceability data collection system for a cloud platform based on multi-layer data fusion, including:

[0086] An information collection module that collects original function call information from each abstraction layer of the cloud platform using the eBPF hook method;

[0087] An eBPF mapping table storage module. For different abstraction layers in the cloud platform, eBPF stores function call information in different key mapping tables according to the characteristics of function calls.

[0088] A trace information optimization module that optimizes trace information for function call information in different abstraction layers;

[0089] A cross-layer information association module that associates the information of different abstraction layers optimized by the trace information optimization module according to the keys and values in different mapping tables to obtain a cross-layer information set;

[0090] A cross-layer traceability graph generation module that generates a cross-layer traceability graph according to the cross-layer information set of the cross-layer information association module.

[0091] This embodiment also provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for collecting fine-grained traceability data of a cloud platform based on multi-layer data fusion.

[0092] Specifically, the above processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0093] Among them, the memory may include a mass storage for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disc, and the like. In a suitable case, the memory may include a removable or non-removable (or fixed) medium. In a suitable case, the memory may be inside or outside the data processing device.

[0094] An embodiment of the present invention also provides a readable storage medium, in which a computer program is stored. The computer program includes program code for controlling a process to execute the process, and the process includes the method for collecting fine-grained traceability data of the cloud platform based on multi-layer data fusion as described above.

[0095] Embodiments of the present invention can be implemented by computer software, which is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware.

[0096] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the scope of the patent of the present invention. For those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. A fine-grained traceability data collection method for a cloud platform based on multi-layer data fusion, characterized in that, It includes the following steps: 1) Use the eBPF hook method to collect the original function call information from each abstraction layer of the cloud platform; 2) For different abstraction layers in the cloud platform, eBPF stores the function call information in different key mapping tables according to the characteristics of the function calls. The key mapping tables include a TCP mapping table, a container orchestration mapping table, a container mapping table, and a file mapping table. The specific sub-steps for eBPF to store the function call information in different mapping tables are as follows: According to the record of the function call, convert the called function name into a string as the characteristic of the function call, and store the function call information in different mapping tables. The mapping table includes a key field and a value field. The key is unique, and a value can be uniquely determined by the key; The key field of the TCP mapping table includes: source IP address and port number; the value field includes destination IP address and port number; the key field of the container orchestration mapping table includes: function and the instance where the function is located; the value field includes container ID and IP address; the key field of the container mapping table includes: container ID; the value field includes the real file name corresponding to the container, process ID pid, specific operation performed, and timestamp; the key field of the file mapping table includes container ID and file name, and the value field includes the mounted file name; 3) Optimize the trace information of the function call information of different abstraction layers; the work of optimizing the trace information of the function call information of different abstraction layers is in the user space, and it includes the following sub-steps: 3.1) Send data from the kernel space to the user space; 3.2) Construct a set V for recording container IDs and a container data set D for collecting data from containers, both of which are set to empty sets, and process each row of the collected data from the key mapping table. If the container ID of the data is already in the set V, it means that the sent data has already recorded the events from this container. At this time, directly extract the container data set D corresponding to this container; if the container ID of the data is not in the set V, skip this data; 3.3) Determine whether the data already exists in the container data set D corresponding to this container. If the data already exists, skip this data. If the data does not exist, add this data to the container data set D; 3.4) Calculate the difference between the timestamp of this data and the timestamp of the earliest data stored in the container data set D. If the difference exceeds one minute, the data of this container will be compressed using the gzip method to obtain an optimized trace information table; 4) Associate the information of different abstraction layers optimized in step 3) according to the keys and values in different mapping tables to obtain a cross-layer information set; 5) Generate a cross-layer traceability graph according to the cross-layer information set.

2. The acquisition method according to claim 1, wherein The work in step 1) is in the kernel space; the abstraction layers of the cloud platform mainly include the physical layer, the operating system layer, the container orchestration layer, and the virtualization layer.

3. The acquisition method according to claim 1, wherein, The association of the information of different abstraction layers optimized in step 3) includes the following sub-steps: 4.1) Input each item in the optimized trace information table as a single data record. Determine whether there is a corresponding data record in the TCP mapping table based on the destination IP address in the TCP mapping table. If not, skip the subsequent steps and directly process the next input data record. If the data record exists, proceed to step 4.2). 4.2) Determine whether the destination IP address is the same as the IP address in the container orchestration mapping table. If the same, add the function instance in the container orchestration mapping table to the cross-layer information set. If different, skip the subsequent steps and return to step 4.1). 4.3) Determine whether the container ID in the container mapping table is the same as the container ID in the container orchestration mapping table, and determine whether the container ID in the file mapping table is the same as the container ID in the container orchestration mapping table. If any one is different, do not store this data record. If all are the same, add the function information and IP address in the container orchestration mapping table to the cross-layer information set for storage. The function information in the cross-layer information set includes the real file name, process ID pid, specific operation performed, timestamp, and file name after mounting. 4.4) Determine whether the function instance has terminated. If it has terminated, proceed to step 5) to generate the cross-layer traceability graph. Otherwise, return to step 4.1) to process the next input data record.

4. The acquisition method according to claim 3, characterized in that, Generating the cross-layer traceability graph based on the cross-layer information set includes the following sub-steps: 5.1) Input the container ID in the cross-layer information set. If it is equal to the container ID for which the traceability graph is to be generated, add the data corresponding to the container ID to the traceability graph data set. The traceability graph data set includes the container traceability graph, function instance traceability graph, function traceability graph, and global traceability graph. 5.2) Repeat step 5.1) until all data is processed. 5.3) Add the function information in the cross-layer information set as nodes in the container traceability graph, and use the function call information as the edges connecting the nodes to generate the container traceability graph. 5.4) First add the container traceability graph to the corresponding function instance traceability graph, then add the function instance traceability graph to the corresponding function traceability graph. Add the function traceability graph to the global traceability graph to obtain the cross-layer traceability graph.

5. A fine-grained traceability data acquisition system for a cloud platform based on multi-layer data fusion for implementing the method according to claim 1, characterized in that Including: An information collection module that collects original function call information from each abstract layer of the cloud platform using the eBPF hook method. An eBPF mapping table storage module. For different abstract layers in the cloud platform, eBPF stores the function call information in different key mapping tables according to the characteristics of the function calls. A trace information optimization module that optimizes the trace information of function call information for different abstract layers. A cross-layer information association module that associates the optimized information of different abstract layers by the trace information optimization module according to the keys and values in different mapping tables to obtain a cross-layer information set. A cross-layer traceability graph generation module that generates a cross-layer traceability graph based on the cross-layer information set of the cross-layer information association module.

6. An electronic device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute the method for collecting fine-grained traceability data of a cloud platform based on multi-layer data fusion according to any one of claims 1 to 5.

7. A readable storage medium, characterized in that, The readable storage medium stores a computer program, and the computer program includes program code for controlling a process to execute the process, and the process includes the method for collecting fine-grained traceability data of a cloud platform based on multi-layer data fusion according to any one of claims 1 to 5.

Images