A Reinforcement Learning-Based Adaptive Policy Generation Method and System for Heterogeneous Resource Scheduling

Abstract

This invention relates to a method and system for generating adaptive strategies for heterogeneous resource scheduling based on reinforcement learning, belonging to the field of cloud computing security. The system comprises a container module, a CVSS database exploitation module, a state mapping module, and a defense environment. The method includes: acquiring all container instances in the current cloud environment and storing them in a container pool; recording the heterogeneous attributes and replica count of each type of container instance; calculating the vulnerability exploitation difficulty and multi-dimensional heterogeneity indicators of the current container pool; and using a reinforcement learning model to determine the defense strategy, inputting the current container pool state into the model, deciding on the defense strategy, and calculating the reward value by weighted summation of the vulnerability exploitation difficulty and heterogeneity indicators of the container pool. This invention comprehensively considers the multi-dimensional heterogeneous attributes and real-time state information of containers, adaptively selecting the optimal defense strategy to reduce defense costs and improve the system's real-time response capability and defense effectiveness.

Description

Technical Field

[0001] This invention relates to the field of cloud computing security technology, and in particular to an adaptive multi-layered defense strategy generation method and system for dynamic security threats in container cloud environments. Background Technology

[0002] With the widespread application of cloud-native technologies in intelligent equipment cluster systems, there is an urgent need for an efficient, secure, and flexible defense solution to address the severe security challenges posed by their openness and dynamism. Currently, Moving Target Defense (MTD) technology, as an effective dynamic defense method, continuously changes the system's attack surface through randomized and diversified strategies, improving its protection against zero-day vulnerability attacks, lateral movement attacks, and container escape attacks.

[0003] However, current MTD (Mean Transmission Decision) technology still has significant limitations: First, existing strategies are typically based on fixed periods or static rules, failing to respond in real time and dynamically adapt to changes in attack patterns, resulting in strategy lag. Second, most technologies are designed for homogeneous environments, failing to fully consider the heterogeneous characteristics of containers and their security impact, leading to poor attack defense against heterogeneous environments. Furthermore, existing technologies frequently switch container or cluster states, resulting in excessive resource overhead and making it difficult to achieve a balance between performance and security. Therefore, a new defense strategy generation method is needed to provide better protection for intelligent equipment cluster systems in container cloud environments. Summary of the Invention

[0004] To address the problems of the aforementioned mobile target defense technologies, such as the inability to respond in real time and dynamically adapt to changes in attack patterns, poor attack defense effectiveness against heterogeneous environments, and excessive resource overhead caused by frequent switching of container or cluster states, this invention provides a method and system for generating adaptive strategies for heterogeneous resource scheduling based on reinforcement learning. By comprehensively considering the multidimensional heterogeneous attributes and real-time state information of containers, the optimal defense strategy is adaptively selected to reduce defense costs and improve the system's real-time response capability and defense effectiveness.

[0005] The method for generating adaptive strategies for heterogeneous resource scheduling based on reinforcement learning of the present invention includes the following steps:

[0006] Step 1: Obtain all container instances in the current system or cloud environment under study and store them in the container pool. Record the heterogeneous attributes and number of replicas for each type of container instance. The attributes include the operating system, CPU architecture, container runtime, and application software type.

[0007] Step 2: Calculate the exploit difficulty and multidimensional heterogeneity index of the current container pool using the CVSS standard. The multidimensional heterogeneity index includes a diversity index and a difference index for the container pool. The diversity index calculates the distribution of all values ​​for each attribute in the current container pool; the difference index calculates the degree of difference for each attribute of container instances within the current container pool. The diversity and difference indices for the four attributes are summed separately and then added together to obtain the heterogeneity index of the current container pool. The exploit difficulty of each container instance is calculated, and then a weighted sum is obtained to obtain the exploit difficulty of the current container pool.

[0008] Step 3: Use reinforcement learning models to make defense strategy decisions and realize adaptive dynamic defense of the system.

[0009] The system state of the reinforcement learning model is the current state of the container pool, which includes the operating system, CPU architecture, runtime environment, application software type, number of replicas, and vulnerability exploitation difficulty of various container instances in the container pool.

[0010] The action space of the reinforcement learning model consists of three defense strategies: container self-cleaning, heterogeneous container switching, and cluster strong switching. The model's action is to select and execute one of these defense strategies. During container self-cleaning, the number of replicas of container instances is randomly adjusted without changing the heterogeneous attributes of the containers. During heterogeneous container switching, a container instance is randomly selected from the current container pool, and at least one heterogeneous attribute of that container instance is updated. During cluster strong switching, a full random switching of heterogeneous attributes is performed simultaneously on all container instances in the current container pool.

[0011] Based on the vulnerability exploitation difficulty ED and heterogeneity index HI of the system or cloud environment container pool after the action is executed, the reinforcement learning model calculates the reward r of the selected action by weighted summation of the calculated ED and HI.

[0012] The trained reinforcement learning model is deployed in the system or cloud environment under study. Based on the real-time collected container pool data, heterogeneity indicators and vulnerability exploitation difficulty are calculated, and the best defense decision is adaptively selected and executed.

[0013] The present invention provides a heterogeneous resource scheduling adaptive policy generation system based on reinforcement learning, comprising a container module, a CVSS (Common Vulnerability Scoring System) database exploitation module, a state mapping module, and a defense environment.

[0014] The container module is used to set the heterogeneous attributes of container instances, the vulnerability exploitation difficulty calculation function, and the attribute update method. The heterogeneous attributes include the operating system, CPU architecture, container runtime, and application software type. There are three attribute update methods: container self-cleaning, heterogeneous container switching, and cluster strong switching. During container self-cleaning, the number of replicas of the container instance is randomly adjusted without changing the heterogeneous attributes of the container. During heterogeneous container switching, a container instance is randomly selected from the current container pool, and at least one heterogeneous attribute of the container instance is updated. During cluster strong switching, a full random switch of heterogeneous attributes is performed simultaneously on all container instances in the current container pool.

[0015] The CVSS database uses a module to obtain the heterogeneous attributes of each container instance, and uses CVSS to calculate the vulnerability exploitation difficulty of each container instance. It sets the method for calculating the vulnerability exploitation difficulty and multi-dimensional heterogeneous indicators for the current container pool. These multi-dimensional heterogeneous indicators include the diversity and difference indicators of the container pool. The vulnerability exploitation difficulty of the current container pool is obtained by weighted summation of the vulnerability exploitation difficulties of all container instances. The diversity indicator calculates the distribution of all values ​​for each attribute in the current container pool. The difference indicator is used to calculate the degree of difference of each attribute of the container instance in the current container pool. The diversity and difference indicators of the four attributes are summed separately and then added together to obtain the heterogeneity indicator HI of the current container pool.

[0016] All container instances in the current research system or cloud environment are retrieved and stored in a container pool. The container pool status includes the heterogeneous attributes, number of replicas, and vulnerability exploitation difficulty of each type of container instance. The status mapping module maps the heterogeneous attributes, number of replicas, and vulnerability exploitation difficulty of each type of container instance in the current container pool into a status vector representation.

[0017] The defense environment utilizes a reinforcement learning model to learn defense strategy decisions, achieving adaptive dynamic defense for the system. The system state of the reinforcement learning model is the current container pool state. The action space of the reinforcement learning model consists of three defense strategies, corresponding to three attribute update methods. The model's action is to select and execute one of the defense strategies. The reinforcement learning model calculates the reward r for the selected action by weighted summing of ED and HI based on the vulnerability exploitation difficulty ED and heterogeneity index HI of the container pool in the system or cloud environment after the action is executed. The trained reinforcement learning model is deployed in the defense environment of the studied system or cloud environment, and the heterogeneity index and vulnerability exploitation difficulty are calculated based on the real-time collected container pool data to adaptively select and execute the best defense decision.

[0018] The advantages and positive effects of this invention are as follows:

[0019] (1) Compared with existing mobile target defense technologies, the method and system of this invention comprehensively consider the impact of the multidimensional heterogeneous attributes of containers and their combination on security, introduce a real-time adaptive decision-making mechanism, significantly reduce defense costs, improve the adaptability to container escape attacks, eliminate the problem of policy lag, effectively improve the system's defense effect and response speed, and ensure the security and reliability of the intelligent equipment cluster system.

[0020] (2) The method and system of this invention construct a heterogeneous container state space for heterogeneous environments, including indicators such as operating system, CPU architecture, container runtime, and application software type. Then, the CVSS standard is used to accurately evaluate the vulnerability exploitation difficulty index of container instances, calculate the vulnerability exploitation difficulty and multi-dimensional heterogeneous indicators of the container pool, and dynamically decide on defense strategies such as container self-cleaning, local heterogeneous container switching, or global cluster strong switching based on the real-time state of the current container pool through reinforcement learning algorithms, ultimately achieving adaptive dynamic defense of the system. This invention comprehensively considers the multi-dimensional heterogeneous attributes and real-time state information of containers, which can reduce defense costs and improve the system's real-time response capability and defense effect. Attached Figure Description

[0021] The exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can more clearly understand the technical features and advantages of the present invention, wherein:

[0022] Figure 1 This is an overall architecture diagram of the heterogeneous resource scheduling adaptive strategy generation system according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram illustrating the dynamic changes in the container state space according to an embodiment of the present invention;

[0024] Figure 3 This is a flowchart illustrating the training process of the reinforcement learning model according to an embodiment of the present invention. Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0026] In the specific embodiments of the present invention, an implementation case of an adaptive reinforcement learning mobile target defense system and method based on heterogeneous containers is given.

[0027] like Figure 1As shown in the embodiments of the present invention, the heterogeneous resource scheduling adaptive strategy generation system based on reinforcement learning mainly includes a container module, a CVSS database exploitation module, a state mapping module, and a defense environment. The container module is used to set the heterogeneous attributes of container instances, the vulnerability exploitation difficulty index calculation function, and the attribute update method. In this embodiment, setting the heterogeneous attributes of container instances includes operating system, CPU architecture, container runtime, and application software type; the attribute update method refers to setting three strategies: container self-cleaning, heterogeneous container switching, and cluster strong switching, to switch the heterogeneous attributes of containers; the vulnerability exploitation difficulty index calculation function is calculated by the CVSS framework for container instances. The container module obtains all container instances in the current system or cloud environment and stores them in a container pool. Container self-cleaning refers to adjusting the number of container instance replicas, refreshing container IPs, and eliminating malicious files injected into containers by attackers without changing the heterogeneous configuration of container instances, thereby reducing the risk of persistent attacks. Heterogeneous container switching refers to randomly selecting a container instance from the current heterogeneous container pool and simultaneously updating one or more heterogeneous attributes of that container instance, such as its operating system, CPU architecture, runtime environment, and application software type. This changes the container's local defense status, effectively disrupting the attacker's target reconnaissance and reducing the probability of a successful attack. Strong cluster switching refers to simultaneously performing a comprehensive random switch of heterogeneous attributes on all container instances in the current container pool. This dynamically changes the overall attack surface of the system, significantly improving the system's overall robustness against attacks.

[0028] The CVSS database utilizes modules to acquire operating system data, CPU data, runtime data, and application software data for each container instance. It then uses a vulnerability exploitation difficulty calculation function to determine the exploitation difficulty of each container instance, thereby calculating the overall exploitation difficulty and multi-dimensional heterogeneous indicators for the entire container pool. This allows for a security risk assessment of the current container pool and quantifies its defense effectiveness. Based on data such as the container instance's operating system, CPU architecture, runtime environment, and application software, the CVSS framework uses a CVE (Common Vulnerabilities and Exposures) database to query the vulnerabilities contained in each container instance. A score is calculated for each vulnerability using the CVSS scoring system, and the exploitation difficulty of the container instance is quantified based on the score. Multi-dimensional heterogeneous indicators include diversity and difference indicators for the container pool, used to evaluate the overall container pool's defense capabilities across multiple heterogeneous attributes.

[0029] The state mapping module is responsible for mapping the specific attributes of the container pool to a state space, enabling the defense environment to make policy decisions. The state mapping module maps the heterogeneous attributes, number of replicas, and exploit difficulty of each container instance in the current container pool to a state vector representation. In this embodiment of the invention, the state mapping module uses one-hot encoding to map each attribute of a container instance to a state vector representation.

[0030] The defense environment manages the state space, action space, reward function, termination condition judgment, and action mapping and policy execution methods. It utilizes a reinforcement learning model to learn defense policy decisions, achieving adaptive dynamic defense. The system state of the reinforcement learning model is the current container pool state; the action space of the model contains three defense policies, corresponding to three attribute update methods; the model's action is to select and execute one of these defense policies; the reinforcement learning model calculates the reward *r* for the selected action by weighted summing of ED and HI based on the vulnerability exploitation difficulty (ED) and heterogeneity index (HI) of the system or cloud environment's container pool after the action is executed; the trained reinforcement learning model is deployed in the defense environment of the studied system or cloud environment, and the heterogeneity index and vulnerability exploitation difficulty are calculated based on real-time collected container pool data, adaptively selecting the best defense decision to execute.

[0031] Based on real-time container state space information and multi-dimensional heterogeneous index values, a dynamic decision is made to select one of the following strategies: container self-cleaning, partial switching of heterogeneous containers, or strong cluster switching.

[0032] like Figure 2 As shown, in the initialization phase of this embodiment of the invention, heterogeneous container instances are deployed on each node of the container cluster, and a pre-configured container resource pool is provided. This resource pool contains all currently existing heterogeneous containers in the studied system or cloud environment. Each heterogeneous container is described by attributes such as operating system type, CPU architecture, runtime environment, and application software. Exemplary configurations of container instances include deploying an Nginx service container running Docker engine on an Ubuntu 22.04 system on an x86 architecture host, or deploying a MySQL database container running containerd on a CentOS 7 system on an AMD architecture host. Based on the action space generated by reinforcement learning decisions, the system can execute three defense strategies: container self-cleaning strategy, heterogeneous container switching strategy, and cluster strong switching strategy. The container self-cleaning strategy randomly adjusts the number of container replicas without changing the heterogeneous attributes of the containers; the heterogeneous container switching strategy randomly selects a single container and updates at least one heterogeneous attribute of that container; the cluster strong switching strategy updates the heterogeneous attributes of all containers simultaneously. After the strategy is executed, the system's state space is updated accordingly to adapt to changes in the attack environment.

[0033] To quantify the heterogeneity of defense resources, this invention proposes a heterogeneity measurement method based on two dimensions: complexity and variability. A container environment consists of multiple microservice sets, each containing multiple heterogeneous containers, each described by attribute tags of different dimensions.

[0034] Let the container cloud environment be represented as follows:

[0035] ;

[0036] in: Represents the entire container cloud environment; element This represents the set of microservices of type i in the environment; n is the number of microservices. The correspondence between microservices and heterogeneous containers, and the relationship between heterogeneous containers and heterogeneous attribute tags, are shown below:

[0037] ; ;

[0038] in: microservices The j-th heterogeneous container instance; microservice There are a total of m heterogeneous container instances; each container instance has multiple "label-value" descriptive items. Each label represents an attribute dimension. Representing heterogeneous containers The label value is assigned in the k-th dimension; the label space dimension d represents the container being heterogeneous at d different levels. In this embodiment of the invention, d is 4, corresponding to four heterogeneous attributes.

[0039] To assess the difficulty of exploiting vulnerabilities in nodes, this embodiment quantifies the exploit difficulty of container instances based on the CVSS 3.1 standard, a common vulnerability scoring system. The exploit difficulty EM of a container instance reflects the ease and technical means of exploiting a vulnerability, and the calculation formula is as follows:

[0040] ;

[0041] Here, AV, AC, PR, and UI represent the attack vector, attack complexity, permission requirements, and user interaction degree of the CVSS base score, respectively.

[0042] Since container environments may contain multiple exploitable vulnerabilities, and attackers' vulnerability exploitation preferences are unknown, it is impossible to accurately determine which vulnerability an attacker will choose to exploit. Therefore, this embodiment proposes an exploit weight (EW) based on the exploit code maturity (ECM) metric in time, to quantify the probability of an attacker exploiting a vulnerability. The exploit difficulty (ED) of the current container environment is obtained by weighted averaging the exploit difficulty of all container instances in the container environment, which can be expressed as:

[0043] ;

[0044] Where V is the container pool set, and v is the v-th type of container instance in the container pool. For each type of container instance, calculate... Then sum them up and divide by the sum of EW to get the exploit difficulty ED. EW is obtained by taking the reciprocal of the CVSS metric E (ExploitCode Maturity), which can also be found in the CVE database.

[0045] The CVSS database utilization module also needs to calculate the diversity and heterogeneity metrics of the current container pool. The container set diversity metric (CI) measures the diversity and evenness of distribution of heterogeneous tags (i.e., heterogeneous attributes) within the container set; the diversity metric is calculated for each attribute tag. The diversity metric for the i-th dimension attribute tag... The distribution of this attribute across all possible values ​​in the current container pool is calculated as follows:

[0046] ;

[0047] in, This represents the proportion of the k-th value of the i-th heterogeneous attribute in the current container pool; It represents the total number of possible values ​​for the i-th heterogeneous attribute, such as the operating system type attribute. It corresponds to four operating system types: Linux, Windows, BSD, and Other. This is the weight of the k-th value of the i-th heterogeneous attribute. The weight is usually 1, but it can be adjusted if certain values ​​need to be emphasized.

[0048] The Dissimilarity Index (DI) of a container collection is used to assess the degree to which security vulnerability threats do not overlap among containers in the collection. Let m be the total number of container instances in the container pool. It is the i-th heterogeneous property of the j-th container instance in the container pool. If is the i-th heterogeneous attribute of the h-th container instance in the container pool, then calculate the difference index of the i-th heterogeneous attribute in the current container pool. as follows:

[0049] ;

[0050] Among them, distance , This represents the difference in the i-th dimension of the attribute between two container instances j and h; This refers to the feature dimension of the i-th attribute, that is, the total number of possible values ​​for the i-th attribute of the container instance. For example, if the operating system type has four options: Linux, Windows, BSD, and Other, then... =4; , Let represent the values ​​of the j-th container instance and the h-th container instance in the s-th feature dimension of the i-th attribute, respectively. The feature values ​​of each dimension of each attribute are normalized to [0,1].

[0051] Therefore, the current heterogeneity index of container pools The calculation is as follows:

[0052] ;

[0053] Among them, CI is a diversity indicator for all attributes. Summing yields the DI variability index across all attributes. Summing up, in this embodiment of the invention, i = 1, 2, 3, 4, which correspond to the four attributes of the container instance, respectively.

[0054] Based on the above system, the method for generating adaptive strategies for heterogeneous resource scheduling based on reinforcement learning implemented in this embodiment of the invention mainly includes the following three steps.

[0055] Step 1: Obtain all container instances in the system or cloud environment under study and store them in a container pool. Record the attributes and replica count of each type of container instance. Attributes include operating system, CPU architecture, container runtime, and application software type. The replica count of a container instance refers to the number of container instances with the same attributes running simultaneously.

[0056] Step 2: Calculate the exploit difficulty and multidimensional heterogeneity metrics of the current container pool using the CVSS standard. Multidimensional heterogeneity metrics include container pool diversity metrics and difference metrics. The specific calculation method is as described above and will not be repeated here.

[0057] Step 3: Use a reinforcement learning model to learn action decisions, with the action being the selected defense strategy.

[0058] Action decision training of the reinforcement learning model in this embodiment of the invention, such as... Figure 3As shown, firstly, the current container pool state space is obtained, including the operating system, CPU architecture, runtime environment, application type, number of replicas, and corresponding defense effectiveness (ED). Then, the reinforcement learning algorithm selects actions based on the current state; the action space includes container self-cleaning, heterogeneous container switching, and cluster strong switching. Next, the defense environment executes the selected actions, updating the container pool state space. After the state update is complete, the system calculates the environment's reward value and feeds it back to the reinforcement learning model. Finally, it determines whether the termination condition has been met: training terminates when the training model tends to converge or reaches the specified number of training iterations. If not, the above loop continues; if it is met, the training process ends.

[0059] In this embodiment of the invention, the reinforcement learning model calculates the reward value of the selected action based on the heterogeneity index HI of the cloud environment after the action is executed and the exploitation difficulty ED. For example, the reward value is calculated as r = α * HI + β * ED, where α and β are weight values ​​set based on experience.

[0060] Deploying the trained reinforcement learning model in the studied system or cloud environment, and calculating heterogeneity indicators and vulnerability exploitation difficulty based on real-time collected container pool data, the model adaptively selects the best defense decision to execute. This adapts to heterogeneous and dynamic environments, reduces defense costs, and improves the system's real-time response capabilities and defense effectiveness.

[0061] Except for the technical features described in the specification, all other technologies are known to those skilled in the art. Descriptions of well-known components and technologies are omitted in this invention to avoid redundancy and unnecessarily limiting the invention. Generally, the various exemplary embodiments disclosed in this invention can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software executed by a controller, microprocessor, or other computing device. When aspects of the embodiments disclosed in this invention are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatus, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0062] The above description is merely one embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating an adaptive strategy of heterogeneous resource scheduling based on reinforcement learning, characterized in that, The application comprises the following steps: Step 1, obtaining all container instances in the current system or cloud environment and storing them in a container pool, recording the heterogeneous attributes and the number of copies of each container instance, and the heterogeneous attributes including the operating system, CPU architecture, container runtime and application software type; Step 2, calculating the vulnerability exploitation difficulty and multi-dimensional heterogeneity index of the current container pool by using the CVSS standard; wherein the multi-dimensional heterogeneity index comprises a diversity index and a difference index of the container pool, and the CVSS represents the Common Vulnerability Scoring System; The vulnerability exploitation difficulty of each container instance is calculated, and the weighted sum of the vulnerability exploitation difficulties of all container instances is obtained to obtain the vulnerability exploitation difficulty ED of the current container pool; The diversity index is used to calculate the distribution of all values of each attribute in the current container pool; the difference index is used to calculate the difference degree of each attribute of the container instance in the current container pool; the diversity index and the difference index of the four attributes are summed and added to obtain the heterogeneity index HI of the current container pool; Step 3, using a reinforcement learning model to learn the defense strategy decision; The system state of the reinforcement learning model is the current container pool state, which includes the operating system, CPU architecture, runtime environment, application software type, number of copies and vulnerability exploitation difficulty of various container instances in the container pool; The action space of the reinforcement learning model is three kinds of defense strategies: container self-cleaning, heterogeneous container switching and cluster strong switching; the action of the model is to select one of the defense strategies to execute; when the container self-cleaning is performed, the number of copies of the container instance is randomly adjusted, and the heterogeneous attributes of the container are not changed; when the heterogeneous container switching is performed, a container instance is randomly selected from the current container pool, and at least one heterogeneous attribute of the container instance is updated; when the cluster strong switching is performed, the heterogeneous attributes of all container instances in the current container pool are simultaneously and comprehensively switched; According to the vulnerability exploitation difficulty ED and the heterogeneity index HI of the container pool of the system or cloud environment after the action is executed, the reinforcement learning model calculates the reward r of the selected action by weighted summing the calculated ED and HI; The trained reinforcement learning model is deployed in the researched system or cloud environment, and the heterogeneity index and the vulnerability exploitation difficulty are calculated according to the real-time collected container pool, and the best defense decision is adaptively selected and executed.

2. The method of claim 1, wherein, In step 2, the general vulnerability scoring system CVSS calculates the vulnerability exploitation difficulty of the current container pool, including: first, according to the attribute data of each container instance and the general vulnerability disclosure database, querying the attack vector score AV, attack complexity score AC, privilege requirement score PR and user interaction score UI of each container instance vulnerability, and calculating the vulnerability exploitation difficulty of the container instance ; querying the attack code maturity index E of the container instance vulnerability from the general vulnerability disclosure CVE database, taking the inverse of E to obtain the vulnerability exploitation weight EW; then calculating the vulnerability exploitation difficulty of the current container pool , wherein V represents the container instance set of the current container pool, and v represents the vth container instance therein.

3. The method of claim 1, wherein, The diversity index of the i-th attribute in step 2 is calculated as: wherein, denotes the total number of possible values of the i-th attribute, denotes the proportion of the k-th value of the i-th attribute in the current container pool, is the weight of the k-th value of the i-th attribute; i = 1, 2, 3, 4, respectively corresponding to the four attributes of the container instance.

4. The method of claim 1, wherein, The difference index of the i-th attribute in step 2 The calculation is as follows: ; wherein m represents the total number of container instances in the current container pool, , respectively, the i-th attribute of the j-th container instance, the i-th attribute of the h-th container instance, the distance , is the characteristic dimension of the i-th attribute, , respectively, the value of the j-th container instance, the value of the h-th container instance at the s-th characteristic dimension of the i-th attribute, the value of each characteristic dimension of each attribute is normalized to [0, 1]; i = 1, 2, 3, 4.

5. The method of claim 1, wherein, In step 3, the reinforcement learning model calculates the reward r of the selected action as r = α * HI + β * ED, and α and β are weight values set according to experience.

6. A reinforcement learning based heterogeneous resource scheduling adaptive strategy generation system, characterized in that, The application comprises the following steps: The container module, the CVSS database utilization module, the state mapping module and the defense environment; wherein: The container module is used to set the heterogeneous attributes of the container instance, the vulnerability exploitation difficulty calculation function and the attribute updating method; the heterogeneous attributes include the operating system, the CPU architecture, the container runtime and the type of application software; the attribute updating method includes three kinds: container self-cleaning, heterogeneous container switching and cluster strong switching; when the container self-cleaning, the number of copies of the container instance is randomly adjusted, and the heterogeneous attributes of the container are not changed; when the heterogeneous container switching, at least one heterogeneous attribute of a container instance is updated by randomly selecting a container instance from the current container pool; when the cluster strong switching, the comprehensive random switching of the heterogeneous attributes is performed on all container instances in the current container pool; The CVSS database utilization module obtains the heterogeneous attributes of each container instance, calculates the vulnerability exploitation difficulty of each container instance by using the CVSS, sets the way of calculating the vulnerability exploitation difficulty and the multi-dimensional heterogeneous index of the current container pool, and the multi-dimensional heterogeneous index includes the diversity index and the difference index of the container pool; the vulnerability exploitation difficulty of all container instances is weighted and summed to obtain the vulnerability exploitation difficulty of the current container pool; the diversity index calculates the distribution of all values of each attribute in the current container pool; the difference index is used to calculate the difference degree of each attribute of the container instance in the current container pool; the diversity index and the difference index of the four attributes are summed and added respectively to obtain the heterogeneity index HI of the current container pool; All container instances in the current researched system or cloud environment are stored in the container pool, and the container pool state includes the heterogeneous attributes, the number of copies and the vulnerability exploitation difficulty of each container instance; the state mapping module maps the heterogeneous attributes, the number of copies and the vulnerability exploitation difficulty of each container instance in the current container pool into a state vector representation; The defense environment utilizes the reinforcement learning model to learn the defense strategy decision, and realizes the adaptive dynamic defense of the system; The system state of the reinforcement learning model is the current container pool state; the action space of the reinforcement learning model is three defense strategies, corresponding to three attribute updating methods; the action of the model is to select one of the defense strategies to execute; the reinforcement learning model calculates the vulnerability exploitation difficulty ED and the heterogeneity index HI of the container pool of the system or cloud environment after the action is executed, and obtains the reward r of the selected action by weighted sum of ED and HI; the trained reinforcement learning model is deployed in the defense environment of the researched system or cloud environment, and the heterogeneity index and the vulnerability exploitation difficulty are calculated according to the real-time collected container pool, and the best defense decision is adaptively selected and executed.

7. The system of claim 6, wherein, The CVSS database utilization module queries the attack vector score AV, the attack complexity score AC, the privilege requirement score PR and the user interaction degree score UI of each container instance vulnerability according to the attribute data of each container instance and the universal vulnerability disclosure database, and calculates the vulnerability exploitation difficulty of the container instance ; queries the attack code maturity index E of the container instance vulnerability from the universal vulnerability disclosure CVE database, takes the inverse of E to obtain the vulnerability exploitation weight EW, and calculates the vulnerability exploitation difficulty of the current container pool , wherein V represents a container instance set of the current container pool, and v represents the vth container instance therein.

8. The system of claim 6, wherein, The CVSS database utilizes a module to calculate the diversity index of the i-th attribute For: ; wherein, represents the total number of possible values of the i-th attribute, represents the proportion of the k-th value of the i-th attribute in the current container pool, is the weight of the k-th value of the i-th attribute; i=1,2,3,4, respectively corresponding to the four attributes of the container instance.

9. The system of claim 6, wherein, The CVSS database utilization module calculates the difference index of the i-th attribute For: ; wherein m represents the total number of container instances in the current container pool, , are the i-th attribute of the j-th container instance and the h-th container instance respectively, the distance , is the feature dimension of the i-th attribute, , are the values of the s-th feature dimension of the i-th attribute of the j-th container instance and the h-th container instance respectively, and the value of each feature dimension of each attribute is normalized to [0, 1]; i=1, 2, 3, 4.

Images