Methods for detecting and managing anomalies impacting a computer environment, devices, and corresponding computer programs

Abstract

Methods for detecting and managing anomalies impacting a computer environment, corresponding computer devices and programs. The present invention relates to a method for training, using reinforced learning, a module for determining a set of corrective actions to be implemented by an environment following the detection of an anomaly impacting it. The method comprises: determining, based on data relating to a state of the environment and a reward whose value is a function of the value of a vector representing said state of the environment, a set of corrective actions offering the highest expected reward value. Figure for the abstract: Fig. 1.

Description

Title of the invention: Methods for detecting and managing anomalies impacting a computer environment, corresponding computer devices and programs. Technical field

[0001] The invention belongs to the general field of telecommunications.

[0002] It finds a privileged but not limiting application in the context of cloud computing.

[0003] The invention relates more particularly to the management of anomalies, or failures, occurring within a cloud-computing environment and in particular their detection as well as the determination of corrective measures intended to be applied within the cloud-computing environment so that the latter returns to normal operation free from anomalies. Previous technique

[0004] Various types of failures can affect a telecommunications network. Some of these failures may be of hardware origin, such as a broken transmission cable, a disconnection between equipment belonging to the telecommunications network, etc. These hardware failures also affect the equipment belonging to the telecommunications network itself and impact its hardware components. Such failures can range, for example, from a faulty power supply to a network card failure or a defective hard drive.

[0005] The advent of 5G (i.e. 5th Generation) compliant telecommunications networks is accompanied by the development and deployment of virtualized functions hosted in servers, or nodes, grouped into clusters, thus giving substance to cloud computing.

[0006] An example of a solution for orchestrating these server clusters is the Kubemetes solution. A Kubemetes node cluster comprises a first node called the management node, or "Kubemetes master", and a plurality of compute nodes, or "Kubemetes node".

[0007] The management node includes a controller, an API module (“Application Programming Interface”) and a database called ETCD which consists of a dynamic register of configuration of the computing nodes.

[0008] A compute node comprises a plurality of containers or "pods". Each container is equipped with resources enabling the execution of one or more tasks. A task, when executed, contributes to the implementation of a network service or function, such as a DHCP (Dynamic Host Configuration Protocol) function, for example.

[0009] The Kubernetes orchestration solution is generally chosen because of its robustness and its ability to handle large-scale deployments. This Kubernetes orchestration solution offers advantages in terms of service deployment and scalability, in particular. However, Kubernetes clusters, or environments, remain susceptible to failures.

[0010] Thus, configuration errors are among the most common anomalies that can affect a Kubernetes environment. They can result from incorrect or poorly structured YAML (Yet Another Markup Language) configuration files. For example, a simple typo in the configuration file can prevent the deployment of a container.

[0011] This type of anomaly can prevent the starting or updating of applications running in the Kubernetes environment, causing service interruptions and / or application unavailability.

[0012] Misconfigured resources can also cause performance problems within the Kubernetes environment. Since the Kubernetes solution relies on the use of quotas to manage resources such as compute resources and / or the memory allocated to each container, if these resources are underestimated, the containers cannot perform the required tasks, and if they are overestimated, this can lead to unnecessary waste of allocated resources.

[0013] Under these conditions, fault management in a Kubernetes environment requires a proactive and continuous approach. Regular monitoring, debugging, and updating of configurations and applications allows for a rapid reduction in the impact of these faults and thus helps maintain the quality of the services provided. To this end, the use of monitoring tools can help detect and correct these faults before they affect end users.

[0014] The detection and prediction of the occurrence of anomalies within a cluster of nodes has been an active area of ​​research for many years, and numerous techniques have been developed for this purpose. Most of these detection and prediction techniques rely on the use of machine learning algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long-term memory models (LSTMs) because of their ability to learn complex patterns from large datasets.

[0015] An example of such a technique is the MAPE solution ("Monitoring, Analysis, Plan and Execute") or MAPE loop. The MAPE loop consists of repeating these four steps within the framework of managing a computer environment.

[0016] Thus, the IT environment is monitored to check for any anomalies. If an anomaly is identified, an analysis is performed to determine its cause. Then, a plan is formulated to resolve the problem, and a solution is implemented at the appropriate time. A MAPE loop, consisting of repeating the various steps, aims to identify quality of service and reliability problems in the IT environment as early as possible and to resolve them effectively.

[0017] However, the application of these monitoring techniques to cloud-computing environments is not entirely satisfactory because it lacks, in particular, a complete framework linking the first phases (Monitoring, Analysis) to the following ones (Planning, Execution).

[0018] There is therefore a need for new solutions to improve the management (detection / prediction, correction) of anomalies occurring in a cloud-computing environment. Description of the invention

[0019] To this end, and according to a first aspect, the invention relates to a method for training by reinforced learning of a module for determining a set of corrective actions intended to be implemented by a computer environment following the detection of an anomaly affecting it, the method, implemented by an electronic device, comprising: - the determination, based on data relating to a state of the environment and a reward whose value is a function of a representative value of a representative vector of said state of the environment, of a set of corrective actions offering the highest expected reward value.

[0020] Correspondingly, the invention also relates to an electronic device configured to train, by means of reinforced learning, a module for determining a set of corrective actions intended to be implemented by a computer environment following the detection of an anomaly affecting it, the method, implemented by an electronic device, comprising at least one processor configured to: - determine, based on data relating to a state of the environment and a reward whose value is a function of a representative value of a representative vector of said state of the environment, a set of corrective actions offering the highest expected reward value.

[0021] It should be noted that the invention is particularly advantageously applicable in the field of cloud computing, but also in the field of electrical power distribution management systems.

[0022] The underlying idea of ​​the invention is that knowing information about the distance between a given state of a computing environment and a normal state—that is, a state free of anomalies—of the same computing environment makes it possible to determine more quickly and efficiently a set of corrective actions to be implemented by the computing environment in order to return to a state as close as possible to its normal state. The invention relies on the use of reinforcement learning, which allows this information to be used to determine the set of actions to be implemented.

[0023] As is known, reinforcement learning consists, for an agent, in learning a set of actions to implement, from experiences, in order to optimize the value of a reward over time.

[0024] To this end, the agent is integrated within an environment and makes decisions based on its current state. In return, the environment provides the agent with a reward, which can be positive or negative. Through iterated experiments, the agent seeks a decisional behavior, also called a policy, which is a function associating the current state with the set of actions to be implemented in order to maximize the sum of rewards over time.

[0025] Classically, reinforcement learning is based on a Markovian Decision Process (MDP), which provides a framework for the problem of learning to achieve a goal. Thus, a problem can be considered a Markovian decision process when it exhibits the following properties: • there exists a finite set of states S of the agent in the environment studied; • there exists a finite set, or game, of actions A that the agent can perform; • There is a set of reward values ​​that the agent can obtain following the implementation, by the environment, of a given set of actions, a reward typically worth 1 when the agent wins and 0 when it loses.

[0026] At each time interval k, the agent perceives its state Sk ∈ S. It perceives a priori the set of possible actions in state Sk, although for simplicity, we can assume that the set of actions is the same in all states. It then chooses an action Ak intended to be implemented by the environment and receives from the latter a new state SLi and a value of the associated reward 7?k+i, which allows the agent to move within the environment.

[0027] Based on the interactions between the agent and the environment, a reinforcement learning algorithm determines a function, called a policy, ir: S → A, which, at each state Sk ∈ S, recommends an action a ∈ A to be executed, which is expected to maximize the total value of the rewards. A policy can also be probabilistic. In this case, the policy jr is expressed as follows: n(a, s) = Pr(ak = dsk = s). What is Pr the probability that the agent chooses to execute action a when the environment is in state 5?

[0028] The idea is then to calculate a priori optimal policy ji* by an iteration of two steps: - evaluation of current policy during which the learning algorithm manipulates a table in which for any state 5 ∘ S, a value from the table indicates whether it is good, according to current policy 1, to be in state s; - improvement of current policy ir by using an approach in which, for each state s, we modify policy 11(5) to choose the action that maximizes the value of the expected reward, according to the values ​​present in the table.

[0029] Since solving a reinforcement learning problem involves evaluating functions (optimal policy or optimal value), it is known to approximate these functions using artificial neural networks that enable deep reinforcement learning. These neural networks then approximate an optimal policy ("Policy Gradient").

[0030] A cloud computing environment such as a Kubemetes environment is a Markovian environment. Indeed, the actions implemented by the environment's orchestration entity depend solely on the current state of the environment. Furthermore, the different states of the environment are discrete states.

[0031] In the present invention, the states of the environment belong to two state subspaces: the subspace of normal states (S) and the subspace of abnormal states (S). Any state belonging to the subspace (■$) reflects the operating condition of an environment that is satisfactory for the operator of that environment. The boundaries of the normal space are determined by threshold values ​​Tg, which are generated with the aid of an autoencoder for each resource group g GG, where G is the set of resources considered to determine a state of the environment, such as, for example: CPU for computing resources, storage for memory resources, and network for network resources.

[0032] Thus, S e S, S e S, and SUS = S.

[0037]

[0033] The vector representing said state of the environment is expressed as follows:

[0034] h= etheP?w

[0035] The coefficients of such a vector represent the average of the squared differences between predicted values ​​and expected values ​​in a data set, here the values ​​of the resources of the environment considered.

[0036] The vector h is determined as follows: Normal if h < Tng V ge G a V ne N Anomaly if age G ane N | h > T g

[0038] where N represents the set of nodes n in the environment executing virtualized network functions such as AMF, UPF, NSSF, UDR, etc.

[0039] In particular modes of implementation of the training process, the value of the reward is a function of a Euclidean norm of the vector representing said state of the environment.

[0040] Such a Euclidean norm is denoted Il Il '

[0041] The use of the Euclidean norm to determine a reward is particularly advantageous because it reflects the deviation of the state sk from a normal state. This allows the module for determining a set of corrective actions to be implemented to more quickly and efficiently determine a new set of corrective actions to be implemented with a higher expected reward value.

[0042] In particular implementation modes of the training method, the value of a reward is inversely proportional to the value of the Euclidean norm of the vector representing the state of the environment.

[0043] Such a reward may be used for any type of cloud-computing environment regardless of the features implemented by the latter and regardless of the type of anomaly impacting the environment, provided that the corrective actions included in the determined corrective action set can be implemented in the environment.

[0044] In particular modes of implementation of the training process, said value of a reward decreases exponentially with the value of the Euclidean norm of the vector representing said state of the environment.

[0045] When a corrective action included in the set of determined corrective actions cannot be implemented in the environment, it must be severely penalized. To this end, the value of the reward decreases exponentially with the value of the Euclidean norm of the vector representing the state of the environment resulting from the non-implementation of this action. Such a reward is used, for example when all the computing resources in the environment are already allocated and the required corrective action is to increase the allocated computing resources.

[0046] In particular modes of implementing the training process, the actions constituting the set of actions belong to the group comprising: - increase or decrease the available computing power of a processor belonging to said environment; - increase or decrease the available memory within the environment; - increase or decrease the bandwidth available for exchanging data with other environments; - increase or decrease the number of virtual machines available within the environment.

[0047] The present solution makes it possible to determine a set of corrective actions which make it possible to play both on the resources allocated to the different containers of the environment but also on the number of containers to be instantiated as well as on the number of virtual machines to be instantiated in each of the containers.

[0048] The invention thus offers a global treatment of anomalies impacting a cloud-computing environment, which offers better responsiveness when an anomaly occurs.

[0049] According to a second aspect, the invention relates to a method for detecting an anomaly impacting a computer environment implemented by an electronic device communicating with said environment, the method comprising: - the detection of an anomaly impacting the environment based on a state of said environment; - the transmission, to a module for determining a set of corrective actions intended to be implemented by the environment, of a request to determine a set of corrective actions including data relating to the state associated with said anomaly and a reward whose value is a function of a value representing a vector representing said state associated with the anomaly.

[0050] Correspondingly, the invention also relates to an electronic device configured to detect an anomaly impacting a computer environment with which it communicates, the electronic device comprising at least one processor configured to: - detect an anomaly impacting the environment based on a state of said environment; - to transmit, to a module for determining a set of corrective actions to be implemented by the environment, a request of determining a set of corrective actions including data relating to the state associated with said anomaly and a reward whose value is a function of a value representing a vector representing said state associated with the anomaly.

[0051] As previously explained in relation to the training method, using a representative value of a vector representing an environmental state to determine a reward is particularly advantageous because it reflects the deviation of the state in question from a normal state. This then allows the module for determining a set of corrective actions to be implemented to determine more quickly and efficiently a new set of corrective actions to be implemented with a higher expected reward value.

[0052] In particular modes of implementation of the detection method, the detection of an anomaly impacting the environment is a function of a current state and a predicted state of said environment.

[0053] Detecting an anomaly based on both a current state and a predicted state of the environment helps to increase the relevance of the determined corrective action plan and the speed of its determination. These aspects are of particular importance depending on the service impacted by the failure or the nature of the failure itself.

[0054] In particular implementation modes of the detection process, the value of the reward is a function of a Euclidean norm of the vector representing said state of the environment.

[0055] In particular modes of implementation of the detection method, when the state associated with the anomaly is the current state, said request for determination of a set of corrective actions includes data relating to said current state.

[0056] In such a case, an anomaly impacts the environment, which needs to be corrected. Therefore, the information transmitted to the module for determining a set of corrective actions relates to the current state of the environment.

[0057] In particular embodiments of the detection method, it further comprises, when the state associated with the anomaly is the predicted state: - the determination of at least one service impacted by said anomaly and / or the severity of said anomaly; and in which, depending on the service impacted and / or the severity of said anomaly, said request for determination of a set of corrective actions includes data relating to said predicted state.

[0058] In such a scenario, an anomaly has not yet impacted the environment, but it could occur if no corrective action is taken. Thus, if such a predicted anomaly impacts a given service or proves to be severe, the information transmitted to the corrective action set determination module concerns the predicted situation of the environment so that the corrective action set determination module determines a set of corrective actions that prevent the occurrence of the predicted anomaly or reduce its impact on the environment.

[0059] In particular modes of implementation of the detection process, the severity of an anomaly being a function of a Euclidean norm of the vector representing the state associated with said anomaly, the data relating to said predicted state are transmitted to the module for determining a set of corrective actions when a value of the Euclidean norm of the vector representing the predicted state is greater than or equal to a threshold.

[0060] Just as is the case for determining the value of a reward, the use of the Euclidean norm to determine the severity of an anomaly is particularly interesting because it reflects the deviation of the considered state (here, the predicted state) of the environment from a normal state. Thus, the higher the value of the norm of the vector representing the state associated with said anomaly, the more severe the predicted anomaly.

[0061] This then allows the module for determining a set of corrective actions to be implemented to determine more quickly and efficiently a new set of corrective actions that prevent the occurrence of the predicted anomaly or reduce its impact on the environment.

[0062] In specific implementation modes of the detection process, the data relating to the predicted state are transmitted to the module for determining a set of corrective actions when the impacted service belongs to a group of services comprising: - eMBB (Enhanced Mobile Broadband) type services; - uRLLC (Ultra Reliable Low Latency Communication) type services; - mMTC (Massive Machine-Type Communications) type services; - V2X (Vehicle to Everything Communication) type services; And - HMTC (High Performance Machine Type) services Communion).

[0063] In particular modes of implementation of the detection method, said value of a reward is inversely proportional to the value of the Euclidean norm of the vector representing said state of the environment.

[0064] In particular modes of implementation of the detection method, said value of a reward decreases exponentially with the value of the Euclidean norm of the vector representing said state of the environment.

[0065] According to a third aspect, the invention relates to a system for managing at least one anomaly impacting a computer environment, the system comprising: - at least one electronic device configured to detect an anomaly impacting said environment according to the invention; and - at least one electronic device configured to train by reinforced learning a module for determining a set of corrective actions according to the invention.

[0066] In a particular embodiment, the processes which are the subject of the invention are implemented by a computer.

[0067] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or more generally in at least one of the electronic devices according to the invention and comprising instructions adapted to the implementation of at least one of the methods as described above.

[0068] Each of these programs can use any programming language, and be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0069] The invention also relates to an information medium or a recording medium readable by a computer, and comprising instructions for a computer program as mentioned above.

[0070] The information or recording medium can be any entity or device capable of storing programs. For example, the medium may include a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a hard disk drive, or a flash memory.

[0071] On the other hand, the information or recording medium can be a transmissible medium such as an electrical or optical signal, which can be transmitted via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0072] The programs according to the invention can in particular be downloaded onto an Internet-type network.

[0073] Alternatively, the information or recording medium may be an integrated circuit in which a program is incorporated, the circuit being adapted to execute or to be used in the execution of the processes according to the invention.

[0074] It can also be envisaged, in other embodiments, that the processes according to the invention, the electronic devices and the anomaly management system according to the invention have in combination all or part of the aforementioned characteristics. Brief description of the drawings

[0075] Other features and advantages of the present invention will become apparent from the description below, with reference to the accompanying drawings, which illustrate an example of an embodiment without being limiting in any way. In the figures:

[0076] [Fig-1] [Fig.1] represents a system for managing at least one anomaly impacting a computer environment conforming to the invention in a particular embodiment;

[0077] [Fig.2] [Fig.2] schematically represents an infrastructure belonging to the system represented in [Fig.1];

[0078] [Fig.3] [Fig.3] schematically represents an orchestration entity belonging to the system represented in [Fig.1];

[0079] [Fig.4] [Fig.4] schematically represents the hardware architecture of a module for detecting an anomaly impacting the IT environment and a module for determining a set of corrective actions to be implemented by the IT environment in a particular mode of implementation;

[0080] [Fig. 5] [Fig. 5] represents, in flowchart form, the main stages of the methods of training by reinforced learning and of detection of an anomaly impacting the computer environment in a particular embodiment mode;

[0081] [Fig.6] Figure 6 represents the space S of possible states of the environment computer science in a particular mode of implementation. Description of the invention

[0082] Fig. 1 represents a system 1 for managing at least one anomaly impacting a computer environment according to the invention in a particular embodiment.

[0083] In this embodiment, system 1 comprises: - at least one monitoring entity 10 of an infrastructure such as, for example, a Kubernetes node cluster, - a determination entity (20) of a set of corrective actions to be implemented within the infrastructure to correct an anomaly, - a MANO orchestration entity for the infrastructure, and - the infrastructure (Infra.)

[0084] Such an Infra infrastructure is represented in more detail with reference to [Fig.2]. Such an Infra infrastructure classically comprises several Container 1-3 containers, a virtualization layer, a data storage entity, computing means, such as one or more processors, and network means with other equipment or other infrastructures.

[0085] A container executes one or more virtualized VNF network functions and includes a virtualized data storage entity "vstorage", means virtualized computing "vcompute" and virtualized communication means "vnetwork" with other containers for example.

[0086] The MANO orchestration entity is shown in more detail with reference to [Fig. 3]. The function of such an orchestration entity is to configure the infrastructure (Infra) according to the tasks that its containers are required to perform. Thus, the MANO orchestration entity reserves the necessary resources in terms of computing power, memory space, number of virtual machines to be instantiated, etc.

[0087] In order to orchestrate, or configure, the Infra infrastructure, the orchestration entity includes, in a known manner, an NFVO (“Network Functions Virtualization Orchestrator”) entity involved in the scaling up and down of resources, a VNFM (“VNF Manager”) entity for managing virtualized VNF network functions, a VIM (“Virtualized Infrastructure Manager”) entity for managing the virtualized infrastructure and finally an SDN controller (“Software-Defined Networking controller”) entity which manages the control of data flows to improve the performance of applications running within the Infra infrastructure.

[0088] The monitoring entity 10, the orchestration entity MANO and the infrastructure Infra constitute a computer environment within the meaning of the present invention.

[0089] The monitoring entity 10 of [Fig.1] includes an anomaly detection module 11 and an anomaly prediction module 12 within the computer environment.

[0090] The detection module 11 is configured to collect information relating to the operation of the infrastructure in order to detect an anomaly. The information collected concerns both the physical layer and the virtualization layer of the infrastructure. Thus, the detection module 11 receives information relating to both the physical and virtualized components of the infrastructure.

[0091] In a manner known per se, the detection module 11 determines a state sk of the computer environment for a time interval k and transmits it to the determination entity 20.

[0092] The anomaly prediction module 12 is configured to collect information relating to the operation of the infrastructure in order to predict the occurrence of a possible anomaly. As with the detection module 11, the information collected concerns both the physical and virtualization layers of the infrastructure. Thus, the prediction module 12 receives information relating to both the physical and virtualized components of the infrastructure.

[0093] In a manner known per se, the prediction module 12 determines a predicted state V of the computing environment for a time interval k and transmits it to the determination entity 20.

[0094] The determination entity 20 of [Fig. 1] comprises a module for detecting an anomaly impacting the IT environment and a module for determining a set of corrective actions to be implemented by the IT environment following the detection of an anomaly impacting it. The operation of these two detection modules 21 and determination modules 22 is described in more detail with reference to [Fig. 5].

[0095] In the embodiment described herein, the detection module 21 and the determination module 22 have the hardware architecture of a computer 30 as illustrated in [Fig. 4]. This hardware architecture includes, in particular, a processor PROC, a random access memory MEM, a read-only memory ROM, a non-volatile memory NVM, and COM communication means enabling the detection module 21 and the determination module 22, in particular, to communicate with each other and with the other components of the system 1. The non-volatile memory NVM constitutes a storage medium according to the invention, readable by the processor PROC and on which programs according to the invention are stored.

[0096] These programs, denoted PROG, are stored in the non-volatile NVM memory and include instructions defining the main steps of the processes according to the invention as implemented by the detection modules 21 and determination modules 22.

[0097] Figure 5 describes the main steps of the methods of training by reinforced learning and of detection of an anomaly impacting the computer environment according to the invention as respectively implemented by the detection module 21 and the determination module 22 in a particular embodiment.

[0098] The steps of the reinforced learning training and anomaly detection processes impacting the IT environment described below illustrate how information relating to the Infra infrastructure collected and processed by entity 10 is transmitted and used by entity 20 so that the latter generates a set of corrective actions which are transmitted to the orchestration entity MANO which is then responsible for reconfiguring the Infra infrastructure in accordance with the actions received.

[0099] More specifically, the detection module 11 and / or the prediction module 12 collect, during a time interval k, data relating to the operation of the infrastructure Infra as discussed with reference to [Fig.1] (S010).

[0100] In a step S020, the detection module 11 of the monitoring entity 10 determines a current state sk of the IT environment and transmits it to the detection module 21 of the determination entity 20.

[0101] In a step S021, the prediction module 12 of the monitoring entity 10 determines a predicted state of the IT environment and transmits it to the detection module 21 of the determination entity 20. Steps S020 and S021 may be concurrent or not depending on the implementations chosen.

[0102] Upon receipt of the information transmitted by the detection module 11 and by the determination module 12, the detection module 21 determines whether an anomaly impacts the IT environment and / or whether an anomaly could impact the IT environment in the near future.

[0103] For this purpose, the detection module 21 determines (S030) the coordinates of a first vector representing the current state of the computer environment and the coordinates of a second vector h(ç j representing the predicted state of the computer environment.

[0104] The vector representing a state of the computer environment is expressed as follows:

[0105] h = (¾) with hgp®, and g GG, where G is the set of resources considered to determine a state of the environment such as, for example: CPU for computing resources, storage for memory resources, and network for network resources

[0106] Figure 6 represents the space S of possible states of the computing environment. This space S of possible states comprises two subspaces: the subspace of normal states (5'), represented by the parallelepiped "Normal State space" in Figure 6, and the subspace of abnormal states (S). In other words, SeS, SgS, and SuS = S.

[0107] Any state sk or Sk belonging to the subspace (S) reflects an operating condition of the computing environment deemed satisfactory by the operator of that computing environment. The boundaries of the subspace (S) are determined by threshold values ​​Tg set, for example, by the operator of the computing environment.

[0108] In other words: Normal if MSEg(k)<Tg VgEGtNnE iN Anomaly ifageGAwe / V| MSE^k) > 7^

[0109] where MSE stands for "Mean Square Error", and N represents the set of nodes n in the computing environment.

[0110] The coefficients m" of the vector h represent the average square of the errors between predicted and expected values ​​of the resources of the computer environment considered. sk(vec mse) = [YES]

[0112] Once the coordinates of the first vector 11(¾) and the second vector 11(¾) have been determined, the detection module 21 calculates the Euclidean norm of each of these two vectors (S040): √(H^) √(H^) and respectively || || • If the Euclidean norm of one of the two vectors 11(¾) or 11(¾)^ rcPæscnlant respectively the current state and the predicted state of the IT environment is greater than or equal to a first threshold (S050) then an anomaly impacts the IT environment if it is the norm of the vector 11(¾) Which is greater than the first threshold, or an anomaly could impact the IT environment in the future if it is the norm of the vector ) Which is greater than the first threshold.

[0113] Such a first threshold can be set by the computer environment operator. Such a first threshold can correspond, for example, to the maximum value that the Euclidean norm of a vector representing a normal state can take.

[0114] In Figure 6, the vector h represents a current state of the computing environment. This current state sk is an abnormal state because the magnitude of the vector h is greater than the first threshold.

[0115] It should be noted that there are other ways than calculating the Euclidean norm of a vector representing a state of the computer environment to detect the occurrence of an anomaly and / or determine its severity.

[0116] In the case where the current state sk of the computer environment is reveals itself to be an abnormal state, meaning that an anomaly is impacting the computer environment, the detection module 21 determines (S060) the value of a reward r(sk) associated with the current state of the computer environment.

[0117] Such a reward sk), whose value can be positive or negative, represents the impact of implementing a set of corrective actions determined by the module of determination 22 and implementation within the IT environment by The MANO entity had an impact on the IT environment.

[0118] In an example implementation of the present solution, the reward r(sk) is determined as a function of the value of the norm || 11(¾) || of the vector hCvA.J- Plus in particular, in a first implementation, the value of the reward r(sk) is inversely proportional to the norm of the vector 11(¾) [|

[0119]

[0120] In a second implementation, the value of the reward ^(¾) decreases exponentially with the value of the norm of the vector h(^) |[

[0121] r(sk) - exp^f'JI

[0122] It should be noted that the value of the reward r(sk) can be a function of metrics other than the value of the Euclidean norm of a vector representing a state of the computing environment.

[0123] Once the value of the reward f (¾ ) is determined, the determination module 21 transmits this value and the vector 11(¾ ) representing the current state of the computer environment to the determination module 22 (S070).

[0124] In the case where the predicted state sk of the IT environment turns out to be an abnormal state, meaning that an anomaly could impact the IT environment in the future, the detection module 21 identifies (S080) the nature of the service or services that could be impacted by the predicted anomaly.

[0125] If it is a priority service, such as for example a uRLLC service or an eMBB service or a V2X service, the detection module 21 implements steps S060 to S070 by calculating the value of a reward ^(¾) associated with the predicted state of the computing environment and transmitting it with the vector h( ) representing the predicted state of the computing environment to the determination module 22.

[0126] If it is a non-priority service, such as an internet data transmission service, the detection module 21 determines the severity of the predicted anomaly (S090) by means of the norm || || of the vector 11(¾) representative of the predicted state.

[0127] If the value of the norm || h(sy) || of the vector representing the predicted state is greater than or equal to a second threshold, the predicted anomaly is considered severe and the detection module 21 implements steps S060 to S070 by calculating the value of a reward r(sk) associated with the predicted state of the computer environment and transmitting it with the vector h( ) representing the predicted state of the computer environment to the determination module 22.

[0128] In a particular implementation, the first and second thresholds may have the same value.

[0129] When the value of the norm || 11(¾) || of the vector J representing the predicted state is less than the second threshold, the predicted anomaly is not considered severe. In this case, the process stops there and resumes at step S010 when information relating to a new current state ^+ie and a new predicted state is collected by the detection module 11 and the prediction module 12 of the monitoring entity 10.

[0130] Upon receipt of the reward value r(sk) or ) determined by the determination module 21 and the state of the corresponding computing environment sk, the determination module 22 proceeds to determine a set of corrective actions, also called intention, intended to be implemented by the MANO orchestration entity in order to correct the current or predicted anomaly.

[0131] The determination module 22 is trained using a reinforcement learning algorithm to determine the set of corrective actions that will have the most significant impact possible on the state of the IT environment, this impact being quantified by means of the value of the reward r.

[0132] The objective of using a set of corrective actions (or intentions) 1¾ is to group a set of actions a to improve the convergence of the reinforcement learning algorithm during training, and thus improve the computing environment recovery time (SRT). The joint implementation of several actions a can reduce the action space A and therefore decrease the complexity of the exploration.

[0133] Such actions consist of: scaling the resources available to a container and scaling containers. These are respectively called vertical scaling and horizontal scaling.

[0134] Each container has an isolated set of virtual resources such as CPU computing power, available memory, a file management system, and communication bandwidth. These resources can be increased or decreased by an amount xs Z for a given container depending on available resources and requirements. There are three possible actions associated with this vertical scaling, such as:

[0135] - none —>0: no change is made;

[0136] - Upscaling +x: Increase the resources of a container by x units;

[0137] - Downscaling x: decrease the resources of a container by x units;

[0138] Av= [+a^,0,-4}

[0139] Scaling allows increasing or decreasing the number of active containers within the computing environment by a quantity y G Z. || There are three possible actions associated with horizontal scaling, such as:

[0140] - none —>0: no change is made;

[0141] - scaling outwards + 4: increase the number of active containers;

[0142] - Scaling inwards; decreasing the number of active containers.

[0143] A* = { + y”, 0, - y"} ng N

[0144] Thus, the space of possible actions A can be expressed as follows:

[0145] A = AvUAa

[0146] For a given state sk or sk of the computer environment, a set of corrective actions may include a single action, for example, or a plurality of actions; in this case, ...,x|},geGand«EÆ.

[0147] A policy G II, where II is the set of possible policies, is one associating a state of the IT environment g £) with at least one action ( a G 1¾ ) intended to be implemented by the orchestration entity MANO. [01481 4s):IF

[0149] In order to determine the set of corrective actions to be implemented by the orchestration entity in response to the detection of an anomaly, the determination module 22 determines (S 100) the 7T* policy that maximizes the expected reward over time.

[0150] Indeed, in a reinforcement learning mechanism, given the dynamic nature of the monitored computing environment, it is important to capture the total expected reward. This makes it possible to measure the overall impact of the choice of a policy on the state of the computing environment and to determine whether a given policy is optimal or not.

[0151] The two functions capture this expected return under a given policy and can be used to determine the set of corrective actions to be implemented by the MANO orchestration entity, namely, the value function and the value-action function a)-

[0152] Such a value-action function expresses the value of an expected reward linked to the implementation of an action a and to following a policy within a computer environment which is in a state s.

[0153] More particularly, the determination of the set of corrective actions and the associated policy tt? which maximize the expected reward over time, is obtained by determining the optimal value-action function Q\s. a)

[0154] \ v „ Q s, a -ris; a v p,^))max Q(s,a)#

[0155] r ^{R,R}

[0156] Given ^s, a), the best policy can be constructed by choosing the action or actions a that satisfy / \ at each state of the system. argmax^s, aj

[0157] Once the value-action function a) has been identified, the set of corrective actions intended to be implemented by the MANO orchestration entity following the detection is determined an anomaly impacting or potentially impacting the IT environment.

[0158] This set of corrective actions is transmitted (SI 10) to the MANO orchestration entity which determines (S 120) a new configuration of the IT environment.

[0159] In another embodiment, the present invention also relates to a method for detecting an anomaly impacting a computer environment implemented by an electronic device communicating with said environment, the method comprising: - the detection of an anomaly impacting the environment based on a current state and a predicted state of said environment; - the transmission, to a module for determining a set of corrective actions intended to be implemented by the environment, of a request to determine a set of corrective actions including data relating to the state associated with said anomaly among the current state and the predicted state and an associated reward.

Claims

Demands

1. Method for training by reinforcement learning of a determination module (22) of a set of corrective actions intended to be implemented by a computer environment following the detection of an anomaly impacting it, the method, implemented by an electronic device, comprising: - the determination (S 100), as a function of data relating to a state of the environment (s ) and a reward (r(¾)) whose value is a function of a value representing a vector (h(szJ) representing said state of the environment, of a set of corrective actions (¼) offering a value of the highest expected reward.

2. Training method according to claim 1 wherein the value of the reward is a function of a Euclidean norm ( || hffyj || ) of the vector (hûfy)) representing said state of the environment.

3. Training method according to claim 2 wherein said value of a reward (^(¾) ) is inversely proportional to the value of the Euclidean norm ( |[ || ) of the representative vector of said state of the environment.

4. Training method according to claim 2 wherein said value of a reward (^(¾) ) decreases exponentially with the value of the Euclidean norm ( || || of the representative vector of said state of the environment.

5. A training method according to any one of claims 1 to 4, wherein the actions (a) constituting the set of actions (1¾) belong to the group comprising: - increasing or decreasing the available computing power of a processor belonging to said environment; - increasing or decreasing the available memory within the environment; - increasing or decreasing the bandwidth available for exchanging data with other environments; - increasing or decreasing the number of virtual machines available within the environment.

6. Method for detecting an anomaly impacting a computer environment implemented by an electronic device communicating with said environment, the method comprising: - the detection (S050) of an anomaly impacting the environment as a function of a state of said environment; - the transmission (SI 10), to a module for determining a set of corrective actions intended to be implemented by the environment, of a request to determine a set of corrective actions including data relating to the state associated with said anomaly and a reward whose value is a function of a value representing a vector representing said state associated with the anomaly.

7. Method for detecting an anomaly according to claim 6, wherein the detection of an anomaly impacting the environment is a function of a current state (s) and a predicted state ( of said environment.

8. Method of detecting an anomaly according to claim 6 or claim 7, wherein the value of the reward / (¾)) is a function of a Euclidean norm ( || hts' / .) || ) of the representative vector of said state of the environment.

9. A method for detecting an anomaly according to any one of claims 7 to 8, wherein, when the state associated with the anomaly is the current state (sJ), said request for determination of a set of corrective actions includes data relating to said current state.

10. A method for detecting an anomaly according to any one of claims 7 to 8, further comprising, where the state associated with the anomaly is the predicted state: - the determination of at least one service impacted by said anomaly and / or the severity of said anomaly; and wherein, depending on the service impacted and / or the severity of said anomaly, said request for determination of a set of corrective actions includes data relating to said predicted state.

11. A method for detecting an anomaly according to claim 10, wherein the severity of an anomaly is a function of a Euclidean norm of the vector representing the state associated with said anomaly, the data relating to said predicted state are transmitted to the determination module (22) of a set of corrective actions when a value of the Euclidean norm of the vector representing the predicted state is greater than or equal to a threshold.

12. A method for detecting an anomaly according to claim 7 or claims 10 or 11, wherein the data relating to said predicted state°(s*)_are transmitted to the determination module (22) of a set of corrective actions when the impacted service belongs to a group of services comprising: - eMBB type services; - uRLLC type services; - rnMTC type services; - V2X type services; and - HMTC type services.

13. Method of detecting an anomaly according to any one of claims 6 to 12, wherein said value of a reward (r (¾ ) ) is inversely proportional to the value of the Euclidean norm ( || 11(¾) || ) vector representing said state of the environment.

14. Method of detecting an anomaly according to any one of claims 6 to 12, wherein said value of a reward (^(¾)) decreases exponentially with the value of the Euclidean norm of the vector ( || 11(¾) [| ) representing said state of the environment.

15. An electronic device configured to detect an anomaly impacting a computer environment with which it communicates, the electronic device comprising at least one processor configured to: - detect an anomaly impacting the environment based on a state of said environment; - transmit, to a determination module (22) of a set of corrective actions intended to be implemented by the environment, a request for

16.

17. determination of a set of corrective actions including data relating to the state associated with said anomaly and a reward whose value is a function of a value representing a vector representing said state associated with the anomaly. An electronic device configured to train, through reinforced learning, a module for determining a set of corrective actions to be implemented by a computer environment following the detection of an anomaly affecting it; the process, implemented by an electronic device, comprising at least one processor configured to: - determine, based on data relating to a state of the environment and a reward whose value is a function of a representative value of a representative vector of said state of the environment, a set of corrective actions offering the highest expected reward value. A system for managing at least one anomaly impacting a computer environment, the system comprising: - at least one electronic device configured to detect an anomaly impacting said environment according to claim 15; and - at least one electronic device configured to train by reinforcement learning a module for determining a set of corrective actions according to claim 16.

Images