Security-aware method and system for GPU cluster

Abstract

The application relates to the technical field of network security perception, in particular to a GPU cluster security perception method and system, which comprises the following steps: collecting multi-source security situation data to provide basic information for subsequent security analysis; constructing and updating a cluster security situation map in real time to update information timeliness and provide strong support for risk assessment; calculating vertex and edge scores by using a dynamic risk assessment model to identify high-risk vertices and key risk transmission paths and provide a basis for resource isolation and scheduling; generating targeted instructions according to security risk scores and a preset security strategy library to effectively cope with different security risks and guarantee the safe and stable operation of the cluster; converting the instructions into specific control commands and distributing and executing the specific control commands to ensure that the security measures are effectively implemented and the cluster security protection capability is improved; and setting a finite state machine to monitor the instruction execution state and automatically trigger a recovery mechanism to guarantee the continuous and safe operation of the cluster.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security perception, and in particular to a GPU cluster security perception method and system. BACKGROUND

[0002] GPU clusters play a crucial role in many fields such as artificial intelligence and scientific computing today, and their efficient and stable operation is the key foundation for the smooth completion of various complex computing tasks.

[0003] However, in the actual application and management of GPU clusters, there are still multiple challenges: traditional methods may only focus on part of the data at the logical or physical layer, and cannot comprehensively obtain multi-source security situation data of each node in the GPU cluster, leading to inaccurate understanding of the overall security situation and difficulty in discovering potential security threats; conventional resource scheduling is mainly based on performance and load factors without fully considering security risks, which may result in high-risk tasks being allocated to inappropriate nodes, increasing the risk of the cluster being attacked; existing security strategies are often static and cannot be dynamically adjusted according to real-time security conditions of the cluster, and when security risks change, effective resource isolation and scheduling instructions cannot be generated in time, resulting in lagging security protection measures; during the execution of resource isolation and scheduling instructions, various abnormal situations may occur, and traditional systems lack effective automatic recovery procedures for abnormal execution states, which can easily lead to system failures and affect the normal operation of the cluster.

[0004] Therefore, the present application proposes a GPU cluster security perception method and system. SUMMARY

[0005] The purpose of the present application is to solve the problems in the background art and propose a GPU cluster security perception method and system.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] A GPU cluster security perception method, comprising:

[0008] S1, collecting multi-source security situation data of each node in the GPU cluster, wherein the multi-source security situation data includes logical layer running data and physical layer environment data;

[0009] S2, based on the multi-source security situation data, constructing and updating a cluster security situation graph in real time, wherein the cluster security situation graph describes the security dependency relationship and risk transmission path between cluster nodes in a graph structure;

[0010] S3, using a dynamic risk assessment model to calculate the security risk score of each vertex and edge in the cluster security situation graph, and identifying high-risk vertices and critical risk transmission paths;

[0011] S4, dynamically generating resource isolation and scheduling instructions according to the security risk score and the preset security policy library, wherein the instructions include task migration instructions for high-risk vertices, logical isolation instructions for critical risk transmission paths, and hardware partition locking instructions for computing tasks;

[0012] S5, executing the resource isolation and scheduling instructions, converting the instructions into GPU cluster control commands, and distributing the commands to corresponding task scheduling entities for execution;

[0013] S6, based on the cluster state change data after instruction execution and real-time multi-source security situation data, incrementally learning and optimizing the dynamic risk assessment model.

[0014] Further, the process of collecting multi-source security situation data of each node in the GPU cluster includes:

[0015] Collecting logical layer running data, including GPU utilization fluctuation variance, video memory access abnormal pattern score, inter-node communication traffic matrix, and software stack integrity measurement data;

[0016] Collecting physical layer environment data, including GPU core area maximum temperature, voltage ripple total harmonic distortion, and power supply ripple spectrum characteristics;

[0017] Timestamp alignment and feature extraction are performed on the collected logical layer and physical layer data to form a security situation feature vector.

[0018] Further, the process of constructing and real-time updating the cluster security situation graph includes:

[0019] Each GPU node is mapped as a vertex in the graph structure, and the physical connection relationship or logical communication relationship between nodes is mapped as an edge;

[0020] Each vertex is assigned an attribute, which includes the logical layer and physical layer security situation feature vector of the node corresponding to the vertex;

[0021] Each edge is assigned an attribute, which includes the encryption strength of the communication link, the historical abnormal communication event frequency, and the bandwidth occupancy rate;

[0022] When the cluster topology or task load changes, add or delete vertices and edges in real time, and update their attributes.

[0023] Further, the process of calculating the security risk score of each vertex and edge in the cluster security situation graph using the dynamic risk assessment model, and identifying high-risk vertices and critical risk transmission paths includes:

[0024] The input of the dynamic risk assessment model is the cluster security situation graph, and the output is the security risk score of each vertex and each edge in the cluster security situation graph;

[0025] The dynamic risk assessment model assesses the endogenous risk of the vertex by analyzing the association attributes of the vertex attributes and the neighborhood vertices and edges in the cluster security situation graph, and assesses the conductive risk between the vertices on the cluster security situation graph; the endogenous risk and the conductive risk are integrated to map the security risk score of each vertex;

[0026] Meanwhile, the dynamic risk assessment model calculates the security risk score of each edge by analyzing the attributes of the edges in the cluster security situation graph; obtains high-risk vertices based on the security risk scores of the vertices; and uses a path search algorithm to identify the path with the greatest risk conduction impact as the key risk conduction path, in combination with the security risk scores of the edges and the topological connection relationship of the cluster security situation graph.

[0027] Further, the process of dynamically generating resource isolation and scheduling instructions according to the security risk score and the preset security policy library includes:

[0028] Different types of computing tasks are preset with differentiated security isolation levels in the security policy library, including at least three levels, and each level is configured with a corresponding security risk threshold, and the third level corresponds to the highest security isolation level;

[0029] When the security risk score of a vertex exceeds the security risk threshold corresponding to the current computing task carried by the corresponding node, a task migration instruction is generated;

[0030] When the key risk conduction path is identified, a logical isolation instruction is generated, and a composite blocking strategy is used to cut off the risk conduction chain;

[0031] For tasks of the highest security isolation level, a hardware partition locking instruction is generated, and critical physical resources are exclusively allocated through GPU hardware virtualization technology.

[0032] Further, the process of executing the resource isolation and scheduling instructions includes:

[0033] The task migration instruction is converted into a resource reservation command of the target node and a task checkpoint creation and transmission command of the source node;

[0034] The logical isolation instruction is converted into a flow table rule or a firewall strategy in a software-defined network;

[0035] The hardware partition locking instruction is converted into a system call to the GPU management program to create or adjust the hardware virtual partition;

[0036] All control commands are distributed to the distributed task scheduling entities including the node controller and the cluster scheduler through a message middleware based on the publish-subscribe mode.

[0037] Further, in the process of executing the resource isolation and scheduling instructions, an automatic recovery process of an abnormal execution state is further included:

[0038] For each issued resource isolation and scheduling instruction, a corresponding finite state machine is set, and the state set of the finite state machine includes: instruction pending, instruction issued pending confirmation, execution in progress, execution success, execution timeout, resource conflict, and node disconnection;

[0039] The state transition is triggered by external events, including feedback messages of the node agent and timeout events of the central monitor; wherein the migration rules include: the instruction is successfully sent to the message middleware, and then the state is changed to issued pending confirmation; the execution signal of the node is received, and then the state is changed to execution in progress; the success code is received, and then the state is changed to execution success; the timeout feedback is not received, and then the state is changed to execution timeout; the resource error code is received, and then the state is changed to resource conflict; the heartbeat is interrupted, and then the state is changed to node disconnection;

[0040] When the finite state machine enters an abnormal state, a predefined recovery mechanism is automatically triggered.

[0041] Further, the process of incremental learning and parameter optimization of the dynamic risk assessment model includes:

[0042] A training framework for model incremental learning is constructed, and the framework uses the cluster state change data collected after the execution of the resource isolation and scheduling instruction as the state feedback of the reinforcement learning environment; a reward function is defined for quantifying the state feedback;

[0043] An online learning mechanism is adopted, and the newly collected multi-source security situation data and the corresponding reward signal are input into the dynamic risk assessment model; the parameter update of the dynamic risk assessment model adopts a proximal policy optimization algorithm.

[0044] The application also provides a security-aware system of a GPU cluster, including:

[0045] A multi-source data acquisition module is configured to acquire multi-source security situation data of each node in the GPU cluster, wherein the multi-source security situation data includes logical layer running data and physical layer environment data;

[0046] A cluster security situation graph construction module is configured to construct and update a cluster security situation graph in real time based on the multi-source security situation data, wherein the cluster security situation graph describes the security dependence relationship and risk transmission path between the cluster nodes in a graph structure;

[0047] A dynamic risk assessment module is configured to calculate the security risk score of each vertex and edge in the cluster security situation graph by using a dynamic risk assessment model, and identify high-risk vertices and critical risk transmission paths;

[0048] A multi-instruction generation module is configured to generate resource isolation and scheduling instructions dynamically according to the security risk score and a preset security policy library, wherein the instructions include task migration instructions for high-risk vertices, logical isolation instructions for critical risk transmission paths, and hardware partition locking instructions for computing tasks.

[0049] An instruction execution and control module is configured to convert the instructions into GPU cluster control commands and distribute the commands to corresponding task scheduling entities for execution.

[0050] A model online learning module is configured to perform incremental learning and parameter optimization on a dynamic risk assessment model based on cluster state change data after instruction execution and real-time multi-source security situation data.

[0051] The system is integrated in the form of an independent coprocessor card in a management node of a GPU cluster or deployed in the form of a software-defined security middleware on each computing node of the cluster.

[0052] Compared with the prior art, the GPU cluster security perception method and system provided by the application have the following beneficial effects:

[0053] The application can comprehensively collect logical layer running data and physical layer environment data, can multi-dimensionally and deeply understand the cluster state, and can provide a basis for subsequent analysis, so that the security assessment is more accurate. The logical layer data reflects the task running and the software condition, and the physical layer data reflects the hardware environment. The application can build and update a cluster security situation map in real time, can intuitively present the security dependence and risk transmission path between nodes in a graph structure, can ensure the timeliness of information by real-time updating, can quickly locate the potential risk propagation path, can prevent risk diffusion in advance, and can improve the overall security defense capability of the cluster. The application can calculate the security risk score of vertices and edges by using a dynamic risk assessment model, can identify high-risk vertices and critical risk transmission paths, can provide a clear direction for resource isolation and scheduling, can preferentially process high-risk areas, and can ensure the safe operation of critical tasks. The application can generate targeted instructions according to the security risk score and a preset strategy, can realize reasonable allocation and isolation of resources, can avoid the influence of high-risk nodes on tasks by task migration, can cut off the risk transmission by logical isolation, and can guarantee exclusive occupation of critical task resources by hardware partition locking, thereby improving the security and stability of the cluster. The application can convert the instructions into specific control commands and distribute the commands for execution, can ensure that the security measures are effectively implemented, and can improve the management efficiency and automation level of the cluster. The application can set a finite state machine to monitor the execution of the instructions, can automatically trigger a recovery mechanism, can enhance the fault tolerance of the system, can reduce manual intervention, and can ensure the continuous and stable operation of the cluster. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The flowchart of the GPU cluster security perception method provided by the application.

[0055] Figure 2 A module diagram of the security-aware method system of the GPU cluster proposed by the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0057] Please refer to Figure 1 The present application provides a security-aware method of a GPU cluster, comprising:

[0058] S1, collecting multi-source security situation data of each node in the GPU cluster, wherein the multi-source security situation data comprises logical layer running data and physical layer environment data;

[0059] S2, constructing and updating a cluster security situation graph in real time based on the multi-source security situation data, wherein the cluster security situation graph describes the security dependency relationship and risk transmission path between the cluster nodes in a graph structure;

[0060] S3, calculating the security risk score of each vertex and edge in the cluster security situation graph by using a dynamic risk assessment model, and identifying high-risk vertices and key risk transmission paths;

[0061] S4, dynamically generating resource isolation and scheduling instructions according to the security risk score and a preset security policy library, wherein the instructions comprise task migration instructions for high-risk vertices, logical isolation instructions for key risk transmission paths, and hardware partition locking instructions for computing tasks;

[0062] S5, executing the resource isolation and scheduling instructions, converting the instructions into GPU cluster control commands, and distributing the commands to corresponding task scheduling entities for execution;

[0063] S6, based on the cluster state change data after the instruction execution and the real-time multi-source security situation data, incrementally learning and optimizing the dynamic risk assessment model.

[0064] It needs to be further explained that, in the specific implementation process, the process of collecting multi-source security situation data of each node in the GPU cluster comprises:

[0065] Collecting logical layer running data, including GPU utilization fluctuation variance, video memory access abnormal mode score, inter-node communication traffic matrix, and software stack integrity measurement data;

[0066] Collecting physical layer environment data, including the highest temperature of GPU core area, voltage ripple total harmonic distortion, and power supply ripple spectrum characteristics;

[0067] Timestamp alignment and feature extraction are performed on the collected logical layer and physical layer data to form a security posture feature vector.

[0068] Specifically, logical layer operation data is collected, including the utilization rate of each GPU computing core sampled periodically at a millisecond interval, and the fluctuation variance of the GPU utilization rate in a preset time window is calculated to form a GPU utilization rate fluctuation variance index. By monitoring the GPU memory controller, the address sequence and timestamp of the memory access are recorded, and a pre-trained anomaly detection model is used to identify and score the access pattern, outputting a memory access anomaly pattern score (where the access pattern identified as different from the normal workload includes but is not limited to high-frequency random access to non-continuous memory addresses or abnormal access attempts to system reserved memory areas). The input features of the anomaly detection model include quantitative indicators extracted from the memory access address sequence, access frequency, and access target area, including continuous address block proportion, address step distribution statistical characteristics, and access address entropy, which are used to represent the locality and randomness of the access pattern. By analyzing the traffic statistics of the cluster interconnection network, a node-to-node communication traffic matrix is constructed with source node, destination node, traffic size, and message type as dimensions. At the same time, through the secure boot chain and trusted execution environment, the software stack integrity measurement data of the current running computing task on the GPU is obtained. The software stack integrity measurement data includes the hash value of the GPU computing kernel, the hash value of the PTX intermediate code or binary code (such as cubin or hsaco file) specific to the computing architecture used to compile the kernel, the digital signature and version information of the GPU runtime library (such as CUDA Runtime or ROCm core library) called by the task, and the hash value of the host driver component key configuration file;

[0069] Collecting physical layer environment data, including obtaining temperature readings at different spatial points inside the chassis by deploying a micro temperature sensor array inside the GPU chip package, near the PCB power management unit, and inside the chassis cooling air duct, and calculating the highest temperature of the GPU core area; sampling the voltage output of the power supply rail through a high-precision ADC circuit, performing Fourier transform on it, analyzing its ripple components and amplitudes at different frequencies, and calculating the voltage ripple total harmonic distortion; by continuously monitoring and analyzing the power supply voltage, extracting its power supply ripple spectrum characteristics, which are used to represent the dynamic physical state of the GPU during operation;

[0070] For all collected logical layer and physical layer data, including GPU utilization fluctuation variance, memory access abnormal pattern score, inter-node communication traffic matrix, software stack integrity metric data, GPU core area maximum temperature, voltage ripple total harmonic distortion degree, and power supply ripple spectrum characteristics, time stamp alignment is performed based on network time protocol, and the aligned data is standardized to form a security posture feature vector with uniform time sequence markers.

[0071] It should be further explained that, in the specific implementation process, the process of constructing and updating the cluster security posture map in real time includes:

[0072] Each GPU node is mapped as a vertex in the graph structure, and the physical connection relationship or logical communication relationship between nodes is mapped as an edge;

[0073] Each vertex is assigned attributes, including the logical layer and physical layer security posture feature vector of the node corresponding to the vertex;

[0074] Each edge is assigned attributes, including the encryption strength of the communication link, the frequency of historical abnormal communication events, and the bandwidth occupancy rate;

[0075] When the cluster topology or task load changes, add or delete vertices and edges in real time, and update their attributes, to maintain the consistency of the cluster security posture map and the real-time state of the cluster;

[0076] Specifically, each physical GPU node in the GPU cluster and each virtual GPU instance divided by virtualization technology is mapped as an independent vertex in the cluster security posture map; the physical connection relationship between nodes (such as high-speed interconnection realized by NVSwitch) is mapped as a physical edge in the cluster security posture map; the logical communication relationship between nodes due to task cooperation (such as data transmission through MPI library) is mapped as a logical edge in the cluster security posture map; each vertex is assigned an attribute set, which is the security posture feature vector of the node corresponding to the vertex; each edge is assigned a dynamic attribute set, which includes the encryption protocol identifier and key length currently used by the communication link as the encryption strength attribute; the frequency of connection replay and data packet verification error events occurring in historical communication on the link is calculated as the historical abnormal event frequency attribute; the average and peak bandwidth occupancy rate of the link in the current time window is calculated as the bandwidth occupancy rate attribute;

[0077] When the cluster topology changes (such as nodes online, offline or failure) or the task load changes, causing new logical communication relationship changes (establishment or old relationship removal), through real-time event listeners, dynamically add and delete vertices and edges in the cluster security situation graph, and immediately update the associated attribute set, so as to ensure that the cluster security situation graph is always highly consistent with the real-time physical layout and logical running state of the GPU cluster, and provides accurate topology structure input for dynamic risk assessment.

[0078] It should be further explained that, in the specific implementation process, the process of calculating the security risk score of each vertex and edge in the cluster security situation graph by using the dynamic risk assessment model and identifying high-risk vertices and key risk transmission paths includes:

[0079] The input of the dynamic risk assessment model is the cluster security situation graph, and the output is the security risk score of each vertex and each edge in the cluster security situation graph;

[0080] Among them, the dynamic risk assessment model analyzes the association attributes of the vertex attributes and the adjacent vertices and edges in the cluster security situation graph to assess the endogenous risk of the vertex, and analyzes the transmission risk between the vertices on the cluster security situation graph; integrate the endogenous risk and the transmission risk to map the security risk score of each vertex;

[0081] At the same time, the dynamic risk assessment model analyzes the attributes of the edges in the cluster security situation graph to calculate the security risk score of each edge; based on the security risk score of each vertex, the high-risk vertices are analyzed; combined with the security risk score of each edge and the topological connection relationship of the cluster security situation graph, the path search algorithm is used to identify the path with the greatest risk transmission impact as the key risk transmission path;

[0082] Specifically, the dynamic risk assessment model is a specially trained graph neural network model, and the model structure adopts a hybrid architecture of graph attention network and graph convolution network; the input of the dynamic risk assessment model is the complete structure data of the cluster security situation graph, including the vertex attribute feature matrix composed of the security situation feature vectors of all vertices, the edge attribute feature matrix composed of the communication and connection attributes of all edges, and the adjacency matrix representing the connection relationship between the vertices in the cluster security situation graph;

[0083] The dynamic risk assessment model learns the attention coefficients of each vertex in the cluster security situation graph to the vertices and connecting edges in its first-order neighborhood through one or more graph attention layers inside the model, so as to dynamically assign different weights to different neighbor vertices and edges in the information aggregation stage, so that the model can focus on the local structure highly related to security risks; at the same time, the dynamic risk assessment model performs message passing and feature propagation on the cluster security situation graph through one or more graph convolution layers to obtain the dependence relationship in the multi-hop range between vertices.

[0084] The output of the dynamic risk assessment model is the security risk score of each vertex and the security risk score of each edge in the cluster security situation graph; wherein the security risk score of each vertex comprehensively considers the endogenous risk determined by the logical layer and physical layer attributes of the vertex itself, and the conductive risk from its neighbor vertices aggregated through graph convolution and attention mechanism; the security risk score of each edge is obtained by analyzing the attributes of the edges in the cluster security situation graph, which represents the potential probability and estimated severity of risk conduction through the communication or connection link;

[0085] In order to identify the key risk conduction path, based on the vertex security risk score output by the dynamic risk assessment model, the vertex security risk score of the vertex exceeding the preset high risk judgment threshold is identified as a high risk vertex, wherein the high risk judgment threshold is used to define the critical value of whether the vertex belongs to a high risk vertex, when the security risk score of the vertex exceeds the threshold, the vertex is identified as a high risk vertex, so as to further analyze the key risk conduction path related to it; based on the security risk score of the edge output by the dynamic risk assessment model and the topology structure of the cluster security situation graph, the path search algorithm (such as the improved Dijkstra algorithm) is used to find out the top K paths connecting any two high risk vertices and the weighted sum of the security risk scores of all edges on the path, and the path is marked as a key risk conduction path;

[0086] The dynamic risk assessment model undergoes an offline supervised pre-training phase; wherein this phase uses labeled historical security situation data for training, the historical security situation data includes confirmed security event data recorded in the historical running of the GPU cluster, and attack data simulated by penetration testing technology; the confirmed security event data is used to label the real risk label for the vertices and edges in the historical cluster security situation graph, and the simulated attack data is used to expand the diversity and complexity of the training samples.

[0087] It needs to be further explained that, in the specific implementation process, according to the security risk score and the preset security policy library, the process of dynamically generating resource isolation and scheduling instructions includes:

[0088] The security policy library presets different security isolation levels for different types of computing tasks, including at least three levels, and each level is configured with a corresponding security risk threshold, and the third level corresponds to the highest security isolation level;

[0089] When the security risk score of a vertex exceeds the security risk threshold corresponding to the current computing task carried by its corresponding node, a task migration instruction is generated;

[0090] When the critical risk conduction path is identified, a logical isolation instruction is generated, and the risk conduction chain is cut off through a composite blocking strategy;

[0091] For tasks of the highest security isolation level, a hardware partition locking instruction is generated, and critical physical resources are exclusively allocated through GPU hardware virtualization technology;

[0092] Specifically, a configurable security policy library is maintained, which presets different security isolation levels for different types of computing tasks based on the data sensitivity attributes of the computing tasks or the security policies of the departments to which the computing tasks belong. The security isolation levels include at least a first level, a second level, and a third level, and each level is configured with a corresponding security risk threshold. The third level is configured with the highest security risk threshold, representing the highest security isolation level;

[0093] When the security risk score of a vertex exceeds the security risk threshold corresponding to the current computing task carried by its corresponding node, a task migration instruction is generated, and the task is migrated to an idle node with a lower security risk score. The instruction specifies the task instance to be migrated, the source node, and one or more candidate nodes selected from the resource pool. The selection criteria are that the current security risk score is lower than the security risk threshold corresponding to the computing task, and the current available computing resources meet the task requirements;

[0094] When the dynamic risk assessment model identifies a critical risk conduction path, a logical isolation instruction is generated. The execution mode of the logical isolation instruction is one of the following or a combination thereof: issuing a flow table rule on a software-defined network controller to limit or cut off communication traffic on the critical risk conduction path; enabling a preset higher level of end-to-end encryption protocol (such as upgrading from AES-128 to AES-256) on the critical risk conduction path; starting content audit and deep packet inspection processes for data packets transmitted via the critical risk conduction path;

[0095] For tasks marked as security level 3, a hardware partition locking instruction is generated. This instruction calls the application programming interface of the GPU hypervisor and uses GPU hardware virtualization technology (such as NVIDIA MIG or AMD MxGPU) to physically isolate and exclusively allocate the specified physical computing cores, memory controller blocks, and on-chip storage resources, forming a protected execution environment. This ensures that the task is not affected by any other tasks on the same GPU during its operation, until the task actively releases the partition or the administrator intervenes to release it.

[0096] It should be further explained that, in the specific implementation process, the execution of resource isolation and scheduling instructions includes:

[0097] Convert the task migration command into a resource reservation command for the target node and a task checkpoint creation and transmission command for the source node;

[0098] Convert logical isolation instructions into flow table rules or firewall policies in software-defined networks;

[0099] Translate hardware partition locking instructions into system calls to the GPU hypervisor to create or adjust hardware virtual partitions;

[0100] All control commands are distributed to the distributed task scheduling entities, including node controllers and cluster schedulers, through a message middleware based on a publish-subscribe pattern.

[0101] Specifically, resource isolation and scheduling instructions are parsed and converted into GPU cluster control commands. For task migration instructions, the parsing process involves extracting the source node identifier, the instance identifier of the task to be migrated, and one or more candidate target node identifiers from the instruction. The conversion process includes: generating and sending a resource reservation command to the first node in the candidate target node sequence. This command is implemented by calling the cluster resource manager's API, explicitly requesting the reservation of a specified number of GPU computing cores, a specific capacity of video memory space, and corresponding process slots. A task checkpoint creation command is generated and sent to the source node. This command triggers the serialization of the task's current memory state, register state, and execution context into a persistent file and stores it in the shared storage system by calling the checkpoint service of the task runtime framework. A task state secure transmission command is generated. This command establishes a TLS-encrypted data channel to transmit the checkpoint file and task metadata from the source node to the target node where resources have been successfully reserved.

[0102] For the logical isolation instruction, the parsing process includes extracting the network address and port information of the source node and the destination node pair involved in the critical risk conduction path to be isolated in the instruction; the conversion process includes generating and sending an OpenFlow flow table modification command to the software-defined network controller, which contains the matching field of matching the network traffic of the above node pair and the corresponding action set, wherein the action set includes the discard action to block the traffic, the action to output to a certain security audit port for deep packet inspection, and the firewall rule update command to generate, which adds or updates the rules to filter the traffic of specific source IP, destination IP and protocol port by calling the firewall management interface on the node operating system;

[0103] For the hardware partition locking instruction, the parsing process includes extracting the physical GPU identifier specified in the instruction and the number of computing units and the memory capacity required for division; the conversion process includes generating a system call command to the underlying GPU management program, which performs the following operations by calling the GPU virtualization management interface provided by the management program: enabling the hardware virtualization partition function on the specified physical GPU; creating or reconfiguring a GPU virtual instance on the physical GPU according to the resource configuration required by the instruction, which exclusively occupies the allocated computing units and memory controllers at the hardware level, forming an execution environment physically isolated from other instances;

[0104] The distribution of all control commands is carried out through a message middleware based on the publish-subscribe mode (such as Apache Kafka or RabbitMQ), wherein the message middleware (such as Apache Kafka or RabbitMQ) sets up an independent topic for each type of instruction; a lightweight agent process deployed on each computing node subscribes to the topic related to its node identifier, receives, parses and executes the arrived commands, and after execution, the agent publishes a state message containing the execution result code (success or error code) and detailed description information to the specified feedback topic, which is aggregated and analyzed by the central monitoring system;

[0105] In the process of executing the resource isolation and scheduling instruction, an automatic recovery process of the abnormal execution state is also included:

[0106] For each resource isolation and scheduling instruction sent out, a corresponding finite state machine is set up, and its state set includes: instruction to be distributed, instruction distributed to be confirmed, execution in progress, execution success, execution timeout, resource conflict and node disconnection;

[0107] The state transition is triggered by external events, including feedback messages of the node agent and timeout events of the central monitor; wherein the migration rules include: the instruction is successfully sent to the message middleware, and then the state is transferred to the distributed to be confirmed; the node confirmation execution signal is received, and then the state is transferred to the execution; the success code is received, and then the state is transferred to the execution success; the timeout is not received feedback, and then the state is transferred to the execution timeout; the resource error code is received, and then the state is transferred to the resource conflict; the heartbeat is interrupted, and then the state is transferred to the node disconnection;

[0108] When the finite state machine enters the abnormal state, a predefined recovery mechanism is automatically triggered;

[0109] Specifically, for the execution process of each distributed resource isolation and scheduling instruction, a corresponding finite state machine instance is created and maintained; wherein the state set of the state machine is defined as: instruction to be distributed, distributed to be confirmed, execution, execution success, execution timeout, resource conflict, node disconnection;

[0110] The state transition is triggered by external events, wherein the external events include feedback message events from the node agent, and timeout timer expiration events set by the central monitor for each state;

[0111] Specific state migration rules include: migrating from instruction to be distributed to distributed to be confirmed, the triggering event is that the control command has been successfully sent to the message middleware; migrating from distributed to be confirmed to execution, the triggering event is that the command has been received and started to be executed in the preset time, and the confirmation signal returned by the node agent is received; migrating from execution to execution success, the triggering event is that the completion signal returned by the node agent is received, and the signal contains a success code; migrating from execution to execution timeout, the triggering event is that no completion or progress feedback signal is received within the preset execution time threshold; migrating from distributed to be confirmed or execution to resource conflict, the triggering event is that the error code returned by the node agent is received, and the code indicates that the resource is insufficient or occupied; migrating from any state waiting for node response to node disconnection, the triggering event is that the heartbeat communication with the node agent is interrupted for more than a set threshold;

[0112] When the state machine enters an abnormal state, a predefined recovery mechanism is automatically triggered: for the execution timeout state, the recovery mechanism includes sending a command cancellation request to the original target node and attempting to redistribute the instruction to the next available node in the candidate node sequence; for the resource conflict state, the recovery mechanism includes rolling back the operation that has been executed on the conflict node, where the rollback operation includes canceling the resources that may have been reserved and immediately retriggering the resource discovery process to find a new feasible target node based on the latest cluster security situation map and resource inventory; for the node disconnection state, the recovery mechanism includes adding the node identification to the isolation list, logically removing it from the current available resource pool, stopping the distribution of any new instructions to it, and generating an alarm event and notifying the cluster management system for further hardware diagnosis and manual intervention.

[0113] Further, in the specific implementation process, the process of incremental learning and parameter optimization of the dynamic risk assessment model includes:

[0114] A training framework for model incremental learning is constructed, which uses the cluster state change data collected after the execution of resource isolation and scheduling instructions as the state feedback of the reinforcement learning environment; a reward function R is defined to quantify the state feedback, where the calculation formula of the reward function is R = w1 x ARk-w2 x PO + w3 x ASOS; in the formula, ARk represents the difference between the security risk scores of the target node before and after executing the instruction; PO is a performance loss indicator calculated by weighting the additional delay time caused by task migration and the idle time of computing resources; ASOS is the overall security situation change calculated based on the statistical distribution change of the risk scores of all vertices in the cluster security situation map; w1, w2, and w3 are preset weight coefficients;

[0115] An online learning mechanism is used to input the newly collected multi-source security situation data and the corresponding reward signal into the dynamic risk assessment model in small batches; the parameter update of the dynamic risk assessment model uses the proximal policy optimization algorithm, which calculates the probability ratio of the current policy and the old policy, combines the advantage function estimation, and uses the clipping mechanism to constrain the step length of policy update in the gradient ascent update process to stabilize the training process;

[0116] To prevent the model from forgetting the historical knowledge obtained in the offline supervised pre-training phase and subsequent online learning during the incremental learning process, a flexible weight consolidation regularization term is introduced into the loss function of the proximal policy optimization algorithm; the construction process of the regularization term includes: before each new round of incremental learning, based on a saved historical data sampling set, the importance measure of each trainable parameter in the model network for historical tasks is calculated, and the importance measure is estimated by calculating the diagonal element of the Fisher information matrix of the trainable parameter on the historical data; wherein the regularization term is in the form of:

[0117] ;

[0118] wherein, represents the flexible weight consolidation regularization term itself, which will be added as an additional term to the total loss function of the model; λ is a preset hyperparameter for controlling the strength of regularization, as a scalar coefficient, used to control the relative strength or influence of the regularization term in the total loss function; is the index of the trainable parameter, used to traverse each trainable parameter in the model; is the importance measure of the th trainable parameter, which is a scalar value, used to quantify the key degree of the parameter for correctly performing the learned historical tasks, and the value is estimated by calculating the diagonal element of the Fisher information matrix of the parameter on the historical data sampling set; represents the value of the th trainable parameter at the present time (i.e. when the update is about to be performed); represents the value of the th trainable parameter after the completion of the last round of incremental learning task; the regularization term limits the change range of the parameter with high importance in the new task learning process by imposing greater punishment on it;

[0119] It can be understood that the entire incremental learning process is asynchronous to the risk assessment service in one or more dedicated model training nodes in the cluster; when the model completes a round of incremental update and passes the validation set evaluation, the old model instance serving in the production environment is replaced by the new model instance through hot switching technology to ensure the continuity of the risk assessment service.

[0120] Referring to Figure 2 , the application provides a GPU cluster security perception method and system, comprising:

[0121] A multi-source data acquisition module is used to acquire multi-source security situation data of each node in the GPU cluster, wherein the multi-source security situation data includes logical layer running data and physical layer environment data;

[0122] a cluster security situation map construction module, configured to construct and update a cluster security situation map in real time based on multi-source security situation data, wherein the cluster security situation map describes security dependency relationships and risk transmission paths among cluster nodes in a graph structure;

[0123] a dynamic risk assessment module, configured to calculate security risk scores of each vertex and edge in the cluster security situation map by using a dynamic risk assessment model, and identify high-risk vertices and key risk transmission paths;

[0124] a multi-instruction generation module, configured to dynamically generate resource isolation and scheduling instructions according to the security risk scores and a preset security policy library, wherein the instructions include task migration instructions for high-risk vertices, logical isolation instructions for key risk transmission paths, and hardware partition locking instructions for computing tasks;

[0125] an instruction execution and control module, configured to convert the instructions into GPU cluster control commands and distribute the commands to corresponding task scheduling entities for execution;

[0126] a model online learning module, configured to perform incremental learning and parameter optimization on the dynamic risk assessment model based on cluster state change data after instruction execution and real-time multi-source security situation data;

[0127] The system is integrated in a management node of a GPU cluster in the form of a separate coprocessor card, or is deployed on each computing node of the cluster in the form of a software-defined security middleware.

[0128] In addition, the formulas involved in the above are calculated by removing the dimension and taking the numerical value, and are obtained by collecting a large amount of data and performing software simulation to obtain a formula closest to the actual situation. The weight coefficients in the formula and the various preset thresholds in the analysis process are set by a person skilled in the art according to the actual situation or obtained by a large amount of data simulation. The size of the weight coefficient is a specific numerical value obtained by quantizing each parameter, which is convenient for subsequent comparison. The size of the weight coefficient depends on the number of sample data and the corresponding processing coefficients preliminarily set by a person skilled in the art for each group of sample data. As long as it does not affect the proportional relationship between the parameters and the quantized numerical values.

[0129] Each embodiment in the specification is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment mainly describes the differences from other embodiments. In particular, for the device embodiment, since it is basically based on the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0130] For convenience of description, the above apparatus is described in various units by function for separate description. Of course, the functions of the units can be implemented in one or more software and / or hardware in implementing the present application.

[0131] Those skilled in the art should understand that embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD ROM, optical storage, etc.) containing computer-usable program code.

[0132] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0133] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0135] Secondly: the drawings of the embodiments disclosed in the present application only involve the structures related to the embodiments disclosed in the present application, and other structures can refer to the general design. In the case of no conflict, the same embodiments and different embodiments of the present application can be combined with each other.

[0136] Finally: the above is only the preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto, any skilled in the technical field within the scope of the present application disclosed by the technical range, according to the technical solution of the present application and its inventive concept to equivalent replacement or change, should be covered within the scope of protection of the present application.

Claims

1. A security awareness method for GPU clusters, characterized by: S1. Collect multi-source security situation data from each node in the GPU cluster. The multi-source security situation data includes logical layer operation data and physical layer environment data. S2. Based on multi-source security situation data, construct and update the cluster security situation map in real time. The cluster security situation map describes the security dependencies and risk transmission paths between cluster nodes in a graph structure. S3. Calculate the security risk score of each vertex and edge in the cluster security situation diagram using the dynamic risk assessment model, and identify high-risk vertices and key risk transmission paths, including: the input of the dynamic risk assessment model is the cluster security situation diagram, and the output is the security risk score of each vertex and each edge in the cluster security situation diagram. Among them, the dynamic risk assessment model evaluates the endogeneous risk of vertices by analyzing the correlation attributes between vertex attributes and neighboring vertices and edges in the cluster security situation map, and analyzes the transmissive risk between vertices on the cluster security situation map; the endogeneous risk and transmissive risk are integrated to map the security risk score of each vertex. Meanwhile, the dynamic risk assessment model calculates the security risk score of each edge by analyzing the attributes of the edges in the cluster security situation diagram; high-risk vertices are obtained based on the security risk scores of each vertex; and the path search algorithm is used to identify the path with the greatest risk transmission impact as the key risk transmission path by combining the security risk scores of each edge with the topological connection relationship of the cluster security situation diagram. S4. Based on the security risk score and the preset security policy library, dynamically generate resource isolation and scheduling instructions, including task migration instructions for high-risk vertices, logical isolation instructions for critical risk transmission paths, and hardware partition locking instructions for computing tasks. S5. Execute resource isolation and scheduling instructions, convert the instructions into GPU cluster control commands, and distribute them to the corresponding task scheduling entities for execution; S6. Based on cluster state change data after instruction execution and real-time multi-source security situation data, incremental learning and parameter optimization are performed on the dynamic risk assessment model.

2. The security awareness method for GPU clusters according to claim 1, characterized in that: The process of collecting multi-source security posture data from each node in the GPU cluster includes: Collect logical layer operation data, including GPU utilization fluctuation variance, memory access anomaly mode score, inter-node communication traffic matrix, and software stack integrity measurement data; Collect physical layer environmental data, including the highest temperature in the GPU core area, voltage ripple total harmonic distortion, and power supply ripple spectrum characteristics; The collected logical and physical layer data are timestamped and features are extracted to form a security situation feature vector.

3. The security awareness method for GPU clusters according to claim 2, characterized in that: The process of building and updating the cluster security posture map in real time includes: Each GPU node is mapped to a vertex in a graph structure, and the physical connections or logical communication relationships between nodes are mapped to edges; Assign attributes to each vertex, where the attributes include the logical and physical security posture feature vectors of the node corresponding to that vertex; Assign attributes to each edge, including the encryption strength of the communication link, the frequency of historical abnormal communication events, and the bandwidth utilization rate. When the cluster topology or task load changes, vertices and edges are added or deleted in real time, and their attributes are updated.

4. The security awareness method for GPU clusters according to claim 1, characterized in that: The process of dynamically generating resource isolation and scheduling instructions based on security risk scores and a pre-defined security policy library includes: In the security policy library, different security isolation levels are preset for different types of computing tasks, including at least three levels, and each level is configured with a corresponding security risk threshold, with the third level corresponding to the highest security isolation level. When the security risk score of a vertex exceeds the security risk threshold corresponding to the current computing task carried by its corresponding node, a task migration instruction is generated. When a key risk transmission path is identified, a logical isolation instruction is generated, and the risk transmission chain is cut off through a composite blocking strategy. For tasks with the highest security isolation level, hardware partition locking instructions are generated, and critical physical resources are exclusively allocated through GPU hardware virtualization technology.

5. The security awareness method for GPU clusters according to claim 4, characterized in that: The process of executing resource isolation and scheduling instructions includes: Convert the task migration command into a resource reservation command for the target node and a task checkpoint creation and transmission command for the source node; Convert logical isolation instructions into flow table rules or firewall policies in software-defined networks; Translate hardware partition locking instructions into system calls to the GPU hypervisor to create or adjust hardware virtual partitions; All control commands are distributed to the distributed task scheduling entities, including node controllers and cluster schedulers, through a message middleware based on a publish-subscribe pattern.

6. The security awareness method for GPU clusters according to claim 5, characterized in that: The execution of resource isolation and scheduling instructions also includes an automatic recovery process for abnormal execution states: For each distributed resource isolation and scheduling instruction, a corresponding finite state machine is set up, whose state set includes: instruction to be distributed, distributed and awaiting confirmation, in execution, execution successful, execution timeout, resource conflict, and node disconnection; State transitions are triggered by external events, including feedback messages from node agents and timeout events from the central monitor. The transition rules include: if the instruction is successfully sent to the message middleware, it transitions to "distributed and awaiting confirmation"; if a node confirmation signal is received, it transitions to "in execution"; if a success code is received, it transitions to "execution successful"; if no feedback is received within a timeout period, it transitions to "execution timeout"; if a resource error code is received, it transitions to "resource conflict"; and if the heartbeat is interrupted, it transitions to "node disconnected". When a finite state machine enters an abnormal state, a predefined recovery mechanism is automatically triggered.

7. The security awareness method for GPU clusters according to claim 1, characterized in that: The process of incremental learning and parameter optimization of a dynamic risk assessment model includes: Construct a training framework for incremental model learning, which uses cluster state change data collected after resource isolation and scheduling instructions are executed as state feedback for the reinforcement learning environment; define a reward function to quantify the state feedback. An online learning mechanism is adopted to input newly collected multi-source security situation data and corresponding reward signals into the dynamic risk assessment model; the parameter update of the dynamic risk assessment model adopts a near-end strategy optimization algorithm.

8. A security awareness system for GPU clusters, characterized in that, The system, which is applied to the security awareness method for GPU clusters as described in any one of claims 1-7, comprises: The multi-source data acquisition module is used to collect multi-source security situation data from each node in the GPU cluster. The multi-source security situation data includes logical layer operation data and physical layer environment data. The cluster security situation map construction module is used to build and update the cluster security situation map in real time based on multi-source security situation data. The cluster security situation map describes the security dependencies and risk transmission paths between cluster nodes in a graph structure. The dynamic risk assessment module is used to calculate the security risk score of each vertex and edge in the cluster security situation map using the dynamic risk assessment model, and to identify high-risk vertices and key risk transmission paths. The multi-instruction generation module is used to dynamically generate resource isolation and scheduling instructions based on security risk scores and a preset security policy library. The instructions include task migration instructions for high-risk vertices, logical isolation instructions for critical risk propagation paths, and hardware partition locking instructions for computing tasks. The instruction execution and control module is used to convert instructions into GPU cluster control commands and distribute them to the corresponding task scheduling entities for execution. The online learning module is used to perform incremental learning and parameter optimization on the dynamic risk assessment model based on cluster state change data after command execution and real-time multi-source security situation data.

9. The system according to claim 8, characterized in that, The system can be integrated into the management node of the GPU cluster as a standalone coprocessor card, or deployed on each computing node of the cluster as a software-defined security middleware.

Images