Containerized micro-service intelligent collaborative arrangement method and device

Abstract

The invention provides a containerized micro-service intelligent collaborative orchestration method and device, and the method comprises the steps: determining a service completion time delay index of a micro-service architecture system, determining a system load balancing index according to the environment information of the micro-service architecture system, and carrying out the orchestration training of an initialization orchestration model according to the index, and obtaining a micro-service orchestration model; inputting the obtained request state information corresponding to the user service request into a micro-service arrangement model for strategy arrangement processing to obtain an arrangement strategy; and deploying the containerized micro-service according to the arrangement strategy, and operating the deployed containerized micro-service to process a user service request. Through application of the method and the device, the technical problems of low service performance of the micro-service architecture system and increased management cost caused by difficulty in balancing two indexes of service completion time delay and system load balance in the micro-service architecture system when the containerized micro-service is arranged by adopting a scaling method in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of microservice technology, and in particular to a containerized microservice intelligent collaborative orchestration method and device. Background Art

[0002] As the complexity of IoT applications increases rapidly, the microservice architecture with higher flexibility has gradually replaced the traditional monolithic architecture. Container technology has become the first choice for microservice deployment due to its lightweight, easy portability, fast startup and secure isolation. However, when containerized services are started, they need to pull and install container images from the image repository, which will cause startup delays. In edge computing with limited bandwidth, its startup delay will significantly affect the service quality. In addition, in the microservice architecture, applications are decoupled into multiple small and autonomous microservices, which form a directed acyclic graph structure with multiple chains through lightweight communication. The completion of microservices needs to consider the execution order and parallel computing, which leads to two optimization goals of containerized microservices, namely service completion delay and system load balancing. Although these optimization goals can be optimized by orchestrating containerized microservices, that is, orchestration can reduce the completion delay of containerized microservices or ensure system load balancing. However, these two optimization goals are contradictory, because only considering the delay may lead to unbalanced system load. Reducing the service completion delay is beneficial to the service user, while ensuring system load balancing is beneficial to the service provider. Therefore, such a contradiction will affect service performance and increase management costs.

[0003] In the prior art, a scalarization method is generally used to convert multiple optimization objectives into a single objective for optimization, thereby designing an intelligent collaborative orchestration method for containerized microservices. However, when optimizing multiple conflicting optimization objectives at the same time, that is, when orchestrating containerized microservices, both application completion delay and system load balancing must be considered. The scalarization method has limitations and the optimization effect is low, and it is impossible to achieve the optimization effect of balancing service completion delay and system load balancing. Summary of the invention

[0004] The present invention provides a method and device for intelligent collaborative orchestration of containerized microservices, which are used to solve the technical problem that when the prior art adopts a scalarization method to orchestrate containerized microservices, it is difficult to balance the two optimization indicators of service completion delay and system load balancing, resulting in poor service performance of the microservice architecture system and increased management costs.

[0005] The present invention provides a containerized microservice intelligent collaborative orchestration method and device, comprising the following steps: Determine a service completion delay index of the microservice architecture system, and determine a system load balancing index of the microservice architecture system based on the environmental information of the microservice architecture system, wherein the service completion delay index is the time required for the microservice architecture system to process a user service request, and the system load balancing index is used to characterize the degree of system load imbalance of the microservice architecture system when processing a user service request; Based on the service completion delay indicator and the system load balancing indicator, the initialization orchestration model is orchestrated and trained to obtain a microservice orchestration model; When the microservice architecture system obtains a user service request, the request status information of the user service request is obtained, wherein the request status information includes a target microservice and a user preference feature, the target microservice is a containerized microservice to be deployed when the microservice architecture system processes the user service request, and the user preference feature is used to characterize the user's preference for the service completion delay indicator and the system load balancing indicator; Inputting the request status information into the microservice orchestration model for strategy orchestration processing to obtain an orchestration strategy for the target microservice; The containerized microservice is deployed according to the orchestration strategy, and the deployed containerized microservice is run through the microservice architecture system to process the user service request.

[0006] In some embodiments, the microservice architecture system includes a terminal layer, an edge layer, and an image layer, the terminal layer includes multiple containerized microservices obtained by splitting user service requests, the edge layer includes multiple edge nodes, the image layer includes multiple container images, and the container image is used to deploy the containerized microservice to the edge node for execution, and determining the service completion delay indicator of the microservice architecture system includes: When the containerized microservice is deployed to the edge node, in the edge node, determining an image pulling time for the containerized microservice to pull a container image from the image layer; After the containerized microservice pulls the container image, determining the service processing time of the edge node processing the containerized microservice, and determining the data transmission time between the edge nodes; A service completion delay index of the microservice architecture system is determined according to the image pulling time, the service processing time, and the data transmission time.

[0007] In some embodiments, the microservice system architecture includes an edge layer, the edge layer includes multiple edge nodes, the environmental information includes the remaining computing resources and the remaining memory of each edge node, and the system load balancing index of the microservice architecture system is determined according to the environmental information of the microservice architecture system, including: For each of the edge nodes, determine a load computing resource according to the remaining computing resource, and determine a load memory according to the remaining memory; Determining a first average value of a load computing resource of the edge node, and determining a second average value of a load memory of the edge node; The total load balancing value of the edge nodes in the microservice architecture system is determined according to the first average value and the second average value, and the average of the total load balancing values ​​is used as the system load balancing indicator of the microservice architecture system.

[0008] In some embodiments, the initialization orchestration model includes a policy network and a judgment network, the training process of the initialization orchestration model is performed according to the training time step, and the initialization orchestration model is orchestrated and trained based on the service completion delay index and the system load balancing index to obtain a microservice orchestration model, including: Acquire environmental status data, wherein the environmental status data includes a target service request and a preference vector, wherein the preference vector is constructed based on a delay weight and a load weight, wherein the delay weight represents a preference of the target service request for the service completion delay indicator, and the load weight represents a preference of the target service request for the system load balancing indicator; At the current training time step, the environment state data is input into the policy network to generate action design data, and the evaluation network is called to predict the environment state data to generate an expected reward vector value for the action design data, wherein the action design data is used to characterize the edge node deployment action, and the edge node deployment action is a deployment action of deploying the containerized microservice of the terminal layer to the edge node in the edge layer when the microservice architecture system processes the target service request, and the current time step is any training time step in the orchestration training process; Constructing a strategy loss value of the initialized orchestration model according to the action design data, and constructing a value function loss value of the initialized orchestration model according to the action design data and the expected reward vector value; The sum of the strategy loss value and the value function loss value is used as the final loss value of the initialized orchestration model in the current training time step, and the initialized orchestration model is trained by the final loss value to obtain a microservice orchestration model.

[0009] In some embodiments, constructing the strategy loss value of the initialization orchestration model according to the action design data includes: Obtaining the prediction probability of the policy network when generating action design data at the current training time step, and determining the policy amplitude value of the policy network between training time steps according to the prediction probability; In the current training time step, the generalized advantage estimation values ​​of the service completion delay index and the system load balancing index are determined respectively; Determine, according to the generalized advantage estimate and the preference vector included in the environmental state data, a delay advantage value of the policy network of the initialization orchestration model for the service completion delay indicator, and a load advantage value of the policy network for the system load balancing indicator; Determining an advantage function value of the strategy network according to the delay advantage value and the load advantage value; A strategy loss value of the initialized orchestration model is constructed according to the advantage function value and the strategy amplitude value.

[0010] In some embodiments, constructing the value function loss value of the initialized orchestration model according to the action design data and the expected reward vector value includes: In the current training time step, the action design data is executed through the microservice architecture system, and the current experience trajectory of executing the action design data is recorded, wherein the current experience trajectory includes the action design data, the preference vector, and the expected reward vector value; Determine the total revenue value of the microservice architecture system for the service completion delay index and the system load balancing index according to the current experience trajectory, and determine the discounted revenue value of the total revenue value; Determine a cumulative reward vector value in a current training time step, and determine an average value of a difference between the cumulative reward vector value and the discounted benefit value; A value function loss value of the initialized orchestration model is constructed based on the average value of the difference and the preference vector.

[0011] In some embodiments, determining the total benefit value of the microservice architecture system for the service completion delay index and the system load balancing index according to the current experience trajectory includes: Obtain the reward vector value of the service completion delay indicator and the system load balancing indicator; Determining a linear reward scalar value according to the preference vector included in the current experience trajectory and the reward vector value; Determine a penalty value obtained by the microservice architecture system after executing the constraint deployment action when executing the action design data corresponding to the current experience trajectory; The difference between the linear reward scalar value and the penalty value is used as the total benefit value of the microservice architecture system for the service completion delay index and the system load balancing index.

[0012] In some embodiments, after recording the experience track of executing the action design data, the method further includes: Storing the current experience trajectory in a historical experience trajectory library, wherein the historical experience trajectory library is used to store the historical experience trajectory recorded by the initialized orchestration model at a previous training time step, and the previous training time step is any training time step before the current training time step; Determine the total benefit value of the microservice architecture system for the service completion delay index and the system load balancing index based on the historical experience trajectory and the current experience trajectory, and construct the value function loss value of the initialization orchestration model based on the total benefit value; Among them, the historical experience trajectory includes historical environment state data, the current experience trajectory includes current environment state data, the historical target service request in the historical environment state data is the same as the current target service request in the current environment state data, and the historical preference vector in the historical environment state data is different from the current preference vector in the current environment state data.

[0013] The present invention also provides a containerized microservice intelligent collaborative orchestration device, which specifically includes the following modules: A determination module is used to determine a service completion delay index of the microservice architecture system, and determine a system load balancing index of the microservice architecture system according to environmental information of the microservice architecture system, wherein the service completion delay index is the time required for the microservice architecture system to process a user service request, and the system load balancing index is used to characterize the degree of system load imbalance of the microservice architecture system when processing a user service request; A training module, used to perform orchestration training on the initialization orchestration model based on the service completion delay indicator and the system load balancing indicator to obtain a microservice orchestration model; An acquisition module, used for acquiring request status information of the user service request when the microservice architecture system acquires the user service request, wherein the request status information includes a target microservice and a user preference feature, the target microservice is a containerized microservice to be deployed when the microservice architecture system processes the user service request, and the user preference feature is used to characterize the user's preference for the service completion delay indicator and the system load balancing indicator; An orchestration module, used to input the request status information into the microservice orchestration model for strategy orchestration processing to obtain an orchestration strategy for the target microservice; A deployment module deploys the containerized microservice according to the orchestration strategy, and runs the deployed containerized microservice through a microservice architecture system to process the user service request.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for intelligent collaborative orchestration of containerized microservices as described in any one of the above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the containerized microservice intelligent collaborative orchestration methods described above.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the containerized microservice intelligent collaborative orchestration methods described above.

[0017] The containerized microservice intelligent collaborative orchestration method and device provided by the present invention first determine the service completion delay index and the system load balancing index of the microservice architecture system, and then perform orchestration training on the initialization orchestration model according to these two indicators to obtain a microservice orchestration model. When there is a user request, the request status information containing the user preference characteristics is input into the microservice orchestration model, so that the microservice orchestration model will generate an orchestration strategy for the containerized microservice according to the user preference characteristics, and then the microservice architecture system can deploy the containerized microservice according to the orchestration strategy. Thus, it is achieved that the containerized microservice orchestration model is used to formulate an orchestration strategy according to the user's preference for the two indicators to reasonably balance the two indicators of the service completion delay index and the system load balancing index, thereby improving the service performance of the microservice architecture system and reducing management costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the containerized microservice intelligent collaborative orchestration method provided by the present invention.

[0020] Figure 2 It is a schematic diagram of the architecture of the microservice architecture system provided by the present invention.

[0021] Figure 3It is a directed acyclic diagram formed by microservices for establishing communication provided by the present invention.

[0022] Figure 4 It is a schematic diagram of the microservice life cycle provided by the present invention.

[0023] Figure 5 It is a training principle diagram of the initialization orchestration model provided by an embodiment of the present invention.

[0024] Figure 6 It is a structural diagram of the containerized microservice intelligent collaborative orchestration device provided by the present invention.

[0025] Figure 7 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Combine the following Figure 1 To describe the containerized microservice intelligent collaborative orchestration method of the present invention, Figure 1 : is a flow chart of the containerized microservice intelligent collaborative orchestration method provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0028] In step 101, a service completion delay index of the microservice architecture system is determined, and a system load balancing index of the microservice architecture system is determined based on environmental information of the microservice architecture system.

[0029] First, the microservice architecture system in the embodiment of the present invention is introduced. Figure 2 As shown, Figure 2 It is a schematic diagram of the architecture of the microservice architecture system provided by the present invention. Figure 2 As shown, the microservice architecture system includes a terminal layer, an edge layer and an image layer. The terminal layer includes multiple containerized microservices obtained by splitting user service requests, the edge layer includes multiple edge nodes, and the image layer includes multiple container images. The container image is used to deploy the containerized microservices to the edge nodes for execution.

[0030] Specifically, the terminal layer is used to accept some delay-sensitive request services randomly generated by users (i.e. Figure 2), these request services S will be disassembled into multiple microservices m, and then deployed to the edge node n of the edge layer for execution. Microservice m needs to rely on the corresponding container C when executing, so it is necessary to pull the image from the image warehouse deployed in the cloud. After the containerized microservice is deployed on the edge node n of the edge layer, the execution of the containerized microservice is completed by pulling the container image. Figure 2 As shown in Figure 1, in the edge layer, edge nodes n are generally distributed near the end users and access the base station through the wireless network within the geographical coverage of the base station. The deployment and execution of containerized microservices at the terminal layer will affect the service completion time indicator, recorded as QoS, and containerized microservices also need to follow some constraints when deploying. The service completion time and constraints will be introduced below and will not be repeated here.

[0031] After the user service request of the terminal is split into multiple microservices, it can be said that the service request is composed of multiple microservices. These microservices may need to frequently transmit data to each other. Therefore, it can be said that there is communication affinity between the two microservices performing data transmission. Of course, a microservice can establish communication affinity with multiple microservices. According to communication affinity, all microservices can be expressed as a directed acyclic graph, such as Figure 3 As shown, Figure 3 It is a directed acyclic graph formed by the microservices for establishing communication provided by the present invention, in which each rectangle represents a microservice m, each directed edge represents communication affinity, and the level of affinity is represented by the width of the edge.

[0032] Based on the above description, the following Figure 1 In step 101, the service completion delay index of the microservice architecture system is first determined to characterize the service completion time (QoS). The service completion delay index is the time required for the microservice architecture system to process user service requests. It is one of the most critical indicators in the microservice architecture system. This indicator characterizes the processing speed of the system to process user service requests and directly affects the user experience.

[0033] The sign of a service request being completed is that all its microservices must be completed, that is, all microservices exiting the directed acyclic graph have been completed. Usually in the directed acyclic graph structure of microservices, some microservices can be executed in parallel, including non-essential data transmission and data processing. Therefore, the service completion time is not a simple accumulation of the completion time and transmission time of each microservice. When calculating the completion time of each microservice, it is necessary to consider the execution sequence and parallel execution.

[0034] See also Figure 4 , Figure 4 This is a schematic diagram of the microservice life cycle provided by the present invention. Figure 4To illustrate the specific process of determining the service completion latency indicator of a microservice architecture system.

[0035] like Figure 4 As shown in the figure, in the microservice architecture system, the complete life cycle of the microservice MS needs to go through the microservice request sending phase (i.e. sending user requests to edge nodes), the container startup phase (container image pulling), the microservice execution phase (data computing), and the data transmission phase (computation data transmission). Here, the data transmission phase exists only when and only when two microservices m with communication affinity are deployed on different edge nodes n.

[0036] First, in the microservice sending phase, generally speaking, the connection between the user and the base station is carried out through an internal network with stable bandwidth. In addition, since the network topology between the edge node and the base station is fixed, in the microservice sending phase, each microservice ( Figure 4 The sending time of the microservices MS1, MS2, ..., MSn shown in the figure is considered to be a constant value, which is not affected by the orchestration strategy and can be ignored. Without loss of generality, it can be uniformly assumed that the sending time of all microservices is a constant .

[0037] The second is the container startup phase. Container startup mainly completes image pulling and image installation from the image repository of the image layer, and is responsible for the preparation of containerized microservices. Container startup occurs after the microservice is scheduled and before the edge node executes. The container shares the operating system kernel, and the installation time is stable at about 1 second regardless of the size of the container image. Therefore, the embodiment of the present invention will not pay attention to the time spent on container image installation.

[0038] Pulling container images will bring about time costs, which is mainly related to the size of the image layer that the edge node needs to pull and the remaining bandwidth of the edge node. When deploying containerized microservices to edge nodes, the image pull time for the containerized microservice to pull the container image from the image layer is determined in the edge node.

[0039] Assume that microservice m belonging to service request s is deployed on edge node n. The required image layer missing from this edge node is represented as a 0-1 matrix, that is, .in, Does the representative need Rady? Layer mirroring, ( ) indicates need, ( ) indicates that it is not needed, so according to Shannon's theorem, the image layer pull time of microservice m The calculation formula is as follows: (1)

[0040] (2)

[0041] In the above formula (2), L represents the number of layers of the pulled container image. Indicates the number of Layer mirroring, Indicates pulling The time required for layer mirroring, j represents the serial number of the container image. C is the channel capacity, and the calculation formula is as shown in the above formula (2). In the above formula (2), S / N represents the signal-to-noise ratio of wireless channel transmission. Indicates the transmission bandwidth.

[0042] At the same time, it is assumed that the containerized microservice edge computing system is a single-threaded download system, that is, only one image layer can be pulled at a time. Therefore, each edge node should maintain an image pull waiting queue A newly arriving containerized microservice container image needs to be added to this queue so that it can only wait for other container images. At the same time, once a layer of image layer is pulled, the layer buffer of the edge node that pulls the layer record is added to the free storage space under the buffer. Therefore, when calculating the pull time of the image layer of microservice m, additional waiting time must be added. , which can be expressed as the following formula (3): (3)

[0043] In the above formula (3), Indicates the pull time of the image layer. Indicates the number of container images waiting in the image pull waiting queue, and i indicates the i-th other container image being waited for.

[0044] Therefore, if microservice m belonging to service request s is scheduled to edge node n, the complete image pull time can be expressed as the following formula (4): (4)

[0045] Finally, the microservice execution phase occurs after the relevant required container images are pulled and installed. After the containerized microservice (a microservice is called a containerized microservice after it has pulled the container image and installed, and can then be deployed to the edge node for execution) is pulled into the container image, the service processing time of the edge node to process the containerized microservice is determined, and the data transmission time between edge nodes is determined.

[0046] Here, in edge node n, the data processing speed of the CPU is mainly related to the host bit number and the CPU main frequency. The mainstream bit number of the current system is 32 bits, which means that 4 bytes of data can be processed in one cycle. The CPU main frequency indicates the number of cycles that can be calculated per second. Assuming that the operating system bit number of all edge nodes n is 32 bits, allocate CPU resources, the data processing speed of the containerized microservice m can be expressed as: , in byte / s. After obtaining the data processing speed, the service processing time of the containerized microservice m can be expressed as the following formula (5): (5)

[0047] In the above formula (5), Indicates the amount of data that edge node n needs to process when processing containerized microservices.

[0048] In addition, the data transmission phase between edge nodes must also be considered, which occurs when and only when there is long-distance data transmission, which is caused by the unfortunate deployment of two containerized microservices with communication affinity on different edge nodes. If containerized microservice m has communication affinity with containerized microservice m' and is respectively scheduled to different edge nodes n and n' for data processing, it means that containerized microservice m can only start execution when and only when the processed data of m' is completely transmitted to arrive. Therefore, the data transmission time between edge nodes must be considered. On the contrary, if they are scheduled to the same edge node, the data transmission time can be ignored due to the huge transmission speed inside the edge node. Therefore, the data transmission time of containerized microservice m can be expressed as the following formula (6): (6)

[0049] In the above formula (6), m and m' represent two containerized microservices with communication affinity, n and n' represent different edge nodes, n = n' means that m and m' are deployed to the same edge node, n ≠ n' means that m and m' are deployed to two different edge nodes, C represents the channel capacity, Indicates the amount of data that needs to be transmitted between containerized microservice m and containerized microservice m'. Represents the set of containerized microservices that have communication affinity with m.

[0050] Finally, according to the mirror pull time , Service processing time and data transfer time , determine the service completion delay indicator of the microservice architecture system , which is the completion time of the service request ( = ) is expressed as the following formula (7): (7)

[0051] In the above formula (7), It represents the completion time of the containerized microservice m' having communication affinity with the containerized microservice m on the edge node n in the service request s.

[0052] Next, based on the environmental information of the microservice architecture system, determine the system load balancing index of the microservice architecture system. The system load balancing index is used to characterize the degree of system load imbalance of the microservice architecture system when processing user service requests, and is calculated based on the current environmental information of the microservice architecture system. Environmental information can be collected and tested based on the actual system environment. The environmental information includes the remaining computing resources and remaining memory of each edge node n in the edge layer. Therefore, for each edge node n, the load computing resources can be determined based on the remaining computing resources, that is, the CPU computing resources used by the edge node, recorded as , and the load memory can be determined based on the remaining memory, that is, the memory used by the edge node, recorded as .

[0053] Next, since there are multiple edge nodes in the edge layer, the first average value of the load computing resources of the edge nodes can be determined, that is, the used CPU computing resources of all edge nodes can be calculated. The average value of The calculation method is to directly accumulate the sum of the CPU computing resources used by all edge nodes, and then divide it by the total number of edge nodes to get the first average value. Next, determine the second average value of the edge node's load memory, that is, calculate the used memory of all edge nodes The average value of The calculation method is to directly accumulate the sum of the CPU computing resources used by all edge nodes, and then divide it by the total number of edge nodes to get the first average value. .

[0054] The final calculation is based on the first average and the second mean Determine the total load balancing value of edge nodes in the microservice architecture system, denoted as Then, the average load balancing value of each edge node is calculated, that is, the mean of the total load balancing value is calculated, and the mean of the total load balancing value is used as the system load balancing index O of the microservice architecture system, which is expressed as the following formula (8): (8)

[0055] In the above formula (8), N represents the total number of edge nodes in the edge layer of the microservice architecture system, and n represents edge node n, that is, the nth edge node.

[0056] In step 102, based on the service completion delay index and the system load balancing index, the initialization orchestration model is orchestrated and trained to obtain a microservice orchestration model.

[0057] Through the above step 101, the service completion delay index and system load index of the microservice architecture system are calculated based on the environmental information. These two indicators are factors that need to be considered and balanced when arranging containerized microservices. Because these two indicators have different focuses for different users, users of the service generally hope for a smaller service processing delay to improve the service experience. Users of the service provider prefer a smaller system load index, that is, to make the system load more balanced, to improve the overall resource utilization of the system and improve the overall performance of the system.

[0058] In the embodiment of the present invention, the orchestration of containerized microservices is to deploy multiple containerized microservices to corresponding edge nodes respectively. Different orchestration methods will lead to different service completion delay indicators and system load indicators. How to formulate a suitable orchestration strategy to deploy containerized microservices to balance these two indicators of the microservice system architecture, that is, to use these two indicators as optimization targets to optimize the needs of different users, is a technical problem to be solved by the embodiment of the present invention.

[0059] Based on this, the embodiment of the present invention optimizes the two optimization goals by designing an initialization orchestration model to formulate a suitable orchestration strategy, but it is important that the orchestration model should give an orchestration strategy for containerized microservices based on balancing these two indicators. Therefore, the embodiment of the present invention adopts a multi-objective reinforcement learning method, takes the service completion delay indicator and the system load balancing indicator as two optimization goals, performs orchestration training on the initialization orchestration model, and obtains a microservice orchestration model, which will be described in detail below.

[0060] First, we introduce the initialization orchestration model. According to the multi-objective reinforcement learning algorithm, the initialization orchestration model includes the strategy network As well as the critic network Critic, the specific structure of the policy network and the critic network can be a multi-layer perceptron. The policy network is responsible for designing the orchestration strategy and generating action design data. The critic network is responsible for evaluating the orchestration strategy designed by the policy network and helping to optimize the policy network during the training process. The training process of the initialization orchestration model is carried out according to the training time step. During each training time step in the training process, a new policy network and a new critic network can be obtained.

[0061] The following describes the training process of the initial arrangement model. For the sake of convenience and clarity, the embodiment of the present invention divides the training process into four steps, from step 1021 to step 1024, and combines Figure 5 To illustrate, Figure 5 It is a training principle diagram of the initialization orchestration model provided by an embodiment of the present invention.

[0062] In step 1021, environmental status data is obtained.

[0063] like Figure 5 As shown, first, it is necessary to obtain training data for training the initialization orchestration model. In this embodiment of the present invention, the environmental status data of the microservice architecture system is used as training data, which is recorded as . Environmental status data Include target service request and the preference vector The target service request is the service request involved in the environment of the microservice architecture system, and the preference vector is the preference set collected from (in, ), each preference vector in the preference set is sampled from Represents the user's service completion delay indicator The preference degree is different from the system load balancing index O. Bias vector Specifically, according to the delay weight and load weight Constructed, delay weight Characterizing the target service request Service completion delay indicator The preference degree, load weight Characterizing the target service request According to the preference degree of the system load balancing index O, the optimization target in the embodiment of the present invention is only the service completion delay index. And the system load balancing index O two indicators, so + =1, so the environmental status data The expression formula is as follows: Formula (9) and Formula (10): (9)

[0064] (10)

[0065] In the above formulas (9) and (10), t represents the current time, That is, it represents the environmental status data at time t.

[0066] Therefore, at different times t, different environmental status data can be collected By designing such environment status data, the initialization orchestration model can learn and understand the actual needs of users through preference vectors, so as to formulate corresponding orchestration strategies according to the needs. Furthermore, the initialization orchestration model can understand that even for the same target server request , the preference vector at different times t This will also lead to service completion delay indicators The different emphases generated by the system load balancing index O ultimately affect the action decision.

[0067] In step 1022, at the current training time step, the environment state data is input into the policy network to generate action design data, and the judgment network is called to predict the environment state data to generate an expected reward vector value for the action design data.

[0068] Obtain environmental status data at multiple times t After that, it can be used to train the initialization orchestration model, such as Figure 5 As shown, the initialization orchestration model given in the embodiment of the present invention is trained according to the training time step T, that is, the initialization orchestration model is updated once for each training time step T. In a current training time step T, the model before the update is called the old strategy model, and the updated model is called the new strategy model.

[0069] During the training process, first at the current training time step T (representing any training time step in the training process), the environment state data Input to the policy network that initializes the orchestration model Generate action design data Specifically, at the current training time step T, the environment state data At the same time, input into the new policy network and the old policy network , where the old policy network is the new policy network in the previous training time step (T-1), and the old policy network First, based on the state environment data Generates the current experience trajectory.

[0070] It should be noted here that if it is the first training time step (T=1), there is no historical experience trajectory and it cannot be sampled. The current experience trajectory can be directly generated or the experience estimate can be randomly initialized for sampling. Here, the historical experience trajectory is the experience trajectory generated after the action design data of the previous training time step (T-1) interacts with the microservice architecture system environment, so it is called the historical experience trajectory, which is expressed as { }, which will be explained in detail below. Based on the status environment data Fit a normal probability distribution function, which outputs the predicted probability of the corresponding choreography action and extracts the predicted probability from the old policy network The experience trajectory corresponding to the sampled choreography action in the generated historical experience trajectory is used to generate the action design data for the current training time step T This action design data It is used to represent the edge node deployment action, which is the microservice architecture system processing the target service request. , the deployment action of deploying the containerized microservice m of the terminal layer to the edge node n in the edge layer, that is, initializing the orchestration model according to the preference vector Developed scheduling strategy.

[0071] In the process of updating the policy network, the old policy network is updated by sampling historical experience trajectories. Parameters in (The update method is to update through the final loss value) to obtain the new strategy network , new strategy network After the parameter update is completed, the updated parameters Pass back to the old policy network After the parameter update is completed, the old policy network after the parameter update is used as the old policy network for the next training time step T+1.

[0072] In the strategic network Generate motion design data At the same time, the initialization orchestration model will call the critic network to evaluate the environmental status data. Make predictions and generate data designed for the action The expected reward vector value of ,in, , Indicates the service completion delay indicator The expected reward vector value of Represents the expected vector value of the system load balancing index O.

[0073] In step 1023, a strategy loss value of the initialized orchestration model is constructed according to the action design data, and a value function loss value of the initialized orchestration model is constructed according to the action design data and the expected reward vector value.

[0074] At any training time step T, initialize the policy network of the orchestration model Generate motion design data , and the judging network Critic evaluates the action design data Conduct evaluation and generate action design data The expected reward vector value of Next, design data according to the action and the expected reward vector Construct the final loss value of the initialized orchestration model in the current training time step T In this embodiment of the present invention, the final loss value of the initialization arrangement model is By Strategy Network The strategy loss value And the loss value of the value function of the judging network Critic of and get.

[0075] In the embodiment of the present invention, according to the action design data Constructing the strategy loss value for initializing the orchestration model , and design data according to actions And the expected reward vector value Construct the value function loss value of the initialization orchestration model , which will be explained below respectively.

[0076] First, we will introduce the design data based on action Constructing the strategy loss value for initializing the orchestration model , obtain the action design data generated by the policy network at the current training time step T The predicted probability Because the new policy network is based on the input state environment data Fit a normal probability distribution function, which outputs the predicted probability of the corresponding choreography action and extracts the predicted probability from the old policy network The experience trajectory corresponding to the sampled choreography action in the generated historical experience trajectory is used to generate the action design data for the current training time step T , so here we select the generated action design data from the multiple predicted probabilities corresponding to the choreography action The predicted probability .

[0077] Next, according to the predicted probability Determine the strategy amplitude value of the strategy network between training time steps T (that is, between the current training time step T and the previous training time step T-1) , so here we need to determine the action design data in the last training time step T-1 The prediction probability of First, based on the state environment data Generate the current experience trajectory. In the process of generating the current experience trajectory, the action design data in the last training time step T-1 will be fitted. The predicted probability can be directly obtained and recorded as , used to determine the strategy amplitude value The determination method is to calculate the current training time step T for the action design data The predicted probability , and the action design data in the last training time step T-1 The predicted probability The probability ratio is used as the strategy amplitude value and the calculation formula is as follows (11): (11)

[0078] Next, in the current training time step T, the generalized advantage estimates of the service completion delay index and the system load balancing index are determined respectively, denoted as According to multi-objective reinforcement learning, here we need to calculate the generalized advantage estimation values ​​of m optimization objectives within a current training time step T. Since the optimization objectives in the embodiment of the present invention are only the service completion delay index and the system load balancing index, the value of m is 2. The generalized advantage estimation value calculation method is as follows: Formula (12) to Formula (14): (12)

[0079] (13)

[0080] (14)

[0081] In the above formulas (12) to (14), represents the expected reward vector value of the optimization target m, t represents the training time step of the unit, and the maximum value is T. is the discount factor, is the discount parameter, is the attenuation coefficient. represents the cumulative reward value of the optimization target m within the training time step T, Represents the time difference error of the optimization target m within the training time step T. r represents the reward gain value of the optimization target m.

[0082] Next, determine the advantage function value of the policy network, first estimate the value of the generalized advantage The preference vector included with the environment state data , determine the delay advantage value of the policy network of the initialization orchestration model for the service completion delay indicator, and the load advantage value of the policy network for the system load balancing indicator. Here, the generalized advantage value of the optimization target is combined with the user's preference vector, so that the policy network can learn the focus of the optimization target according to the preference vector. Here, the generalized advantage estimate can be weighted directly by the preference vector. Therefore, the delay advantage value of the policy network of the initialization orchestration model for the service completion delay indicator is , the load advantage value of the policy network for the system load balancing index is .

[0083] Then, according to the delay advantage value and the load advantage value, the advantage function value of the strategy network is determined. , the calculation formula can be expressed as the following formula (15):

[0084] In the above formula (15), N represents the total number of preference vectors, i represents the i-th preference vector, and M represents the total number of optimization targets. In the embodiment of the present invention, the optimization targets are only the service completion delay index and the system load balancing index. Then, M takes a value of 2, and m represents the m-th optimization target.

[0085] Finally, according to the advantage function value And the strategy amplitude value Construct the strategy loss value of the initialized orchestration model The calculation formula is as follows (16): (16)

[0086] In the above formula (16), min represents the minimum value function, clip represents the limit value range function, Represents the learning rate, so when the policy network updates the parameters, the parameters Updated to .

[0087] The following describes the specific process of constructing the value function loss value of the initialization orchestration model based on the action design data and the expected reward vector value.

[0088] Since the critic network Critic is used to evaluate the action design data generated by the policy network, the evaluation method is to evaluate the action design data generated by the policy network at the current training time step T. Put it back into the microservice architecture system environment for execution verification, then record the current experience trajectory according to the verification process, and then calculate the total benefit value of the optimization target , used for action design data Evaluate and then construct the value function loss value .

[0089] Therefore, the calculated value function loss value The first step is to execute action design data through the microservice architecture system within the current training time step , and record the execution action design data The current experience trajectory includes the action design data , preference vector w, expected reward vector value Parameters such as .

[0090] The execution process can be that each unit training time step (moment) t is based on the environment state data , then select Action Design Data , then at the next unit training time step t+1, the environment of the microservice architecture system is based on the transition probability p ( | , )∈p execution and transfer its state to the next state At the same time, execute the action design data After that, you will get instant feedback, that is, you can get the execution action design data from the environment of the microservice architecture system based on the expected reward vector value. Total revenue after .

[0091] Calculate the loss value of the value function The second step is: according to the current experience trajectory, determine the total benefit value of the microservice architecture system for the service completion delay indicator and the system load balancing indicator, and determine the discounted benefit value of the total benefit value. The current experience trajectory here records the corresponding preference vector w, so the benefit value of the corresponding indicator can be calculated according to the preference vector w of the corresponding indicator.

[0092] The specific process is to first obtain the reward vector value of the service completion delay indicator and the system load balancing indicator. , this reward vector value can be preset and expressed as the following formula (17) and formula (18): (17)

[0093] (18)

[0094] In the above formulas (17) and (18), Indicates the service completion delay indicator The revenue value, Represents the benefit value of the system load balancing indicator O.

[0095] Next, according to the preference vector w and reward vector value included in the current experience trajectory , determine the linear reward scalar value. Similarly, here we can directly calculate the reward vector value by weighted multiplication through the preference vector. The linear reward scalar value is recorded as It is expressed as the following formula (19): (19)

[0096] Then determine the penalty value obtained by the microservice architecture system after executing the constraint deployment action when executing the action design data corresponding to the current experience trajectory In multi-objective reinforcement learning, the benefit value of the optimization objective is used to guide learning how to generate action design data and perform execution deployment.

[0097] However, when executing the actual deployment action of the containerized microservice, some constraints (or restriction rules) may be violated. The consequences of these constraints will lead to a decrease in the corresponding benefit value. Therefore, the action that violates the constraint in the action design data becomes a constrained deployment action. Therefore, in the embodiment of the present invention, a penalty is imposed on the action design data action that violates the constraint, and the penalty value is obtained. , and the penalty value Add to revenue value middle.

[0098] The present invention divides constraints into soft constraints and hard constraints. Specifically, in the process of scheduling containerized microservices, some containerized microservices must be deployed, which is called constraint 1. If the necessary containerized microservices are not deployed, constraint 1 is violated. In addition, the deployed edge nodes cannot exceed the resource capacity of the edge nodes, which is called constraint 2. If it exceeds, constraint 2 is violated. Both constraints 1 and 2 become hard constraints, that is, violations are prohibited. In theory, unnecessary containerized microservices cannot be deployed, which is called constraint (3). If unnecessary containerized microservices are deployed, constraint 3 is violated. However, in the case of violating constraint 3, the integrity of the microservice is guaranteed, which is acceptable to some extent and is called a soft constraint. Obviously, violations of hard constraints and soft constraints should have different degrees of penalties. A penalty quota is introduced in the embodiment of the present invention. , and the penalty level for violating the hard constraint Set as the penalty level for violating soft constraints Therefore, the penalty value It can be expressed as the following formula (20): (20)

[0099] In the above formula (20), represents the penalty level for violating hard constraints, represents the number of violations of the hard constraint, represents the penalty level for violating the soft constraint, represents the number of violations of soft constraints, Represents the preset penalty factor.

[0100] Finally, the linear reward scalar value With penalty value The difference is used as the total benefit value of the microservice architecture system for the service completion delay index and the system load balancing index , expressed as the following formula (21): (twenty one)

[0101] Calculate the total revenue Then determine the total revenue value The discounted revenue value is directly weighted by the discount factor, and the discounted revenue value is expressed as ,in, represents the discount parameter, represents the discount factor, and m represents the mth optimization objective.

[0102] Calculate the loss value of the value function The third step is to determine the cumulative reward vector value in the current training time step and determine the average value of the difference between the cumulative reward vector value and the discounted benefit value.

[0103] Here, the cumulative reward vector value within the current training time step T can be obtained by the expected reward vector value included in the current experience trajectory To calculate, then calculate the cumulative reward vector value and the discounted benefit value The difference between the two is taken as the cumulative profit loss of the optimization target in the current training time step T. Finally, the average value of the cumulative profit loss is calculated as the profit loss in the unit training time step t, recorded as , which is specifically expressed as the following formula (22): (twenty two)

[0104] Calculate the loss value of the value function The last step is to integrate the preference vector w into the profit loss to guide the importance of the critic network for different optimization goals. The integration process is to directly use the preference vector The profit loss is weighted, so the final value function loss is expressed as the following formula (23): (twenty three)

[0105] Then the value function loss value The calculation method of is finally expressed as the following formula (24): (twenty four)

[0106] In the above formulas (23) and (24), represents the discount parameter, represents the discount factor, m represents the mth optimization objective, The i-th preference vector in the table, N represents the number of preference vectors, M represents the number of optimization targets, which is 2 in the embodiment of the present invention, T represents the current training time step, t represents the unit training time step, and the maximum value is T.

[0107] In some embodiments, Figure 5 As shown, in the calculation value function loss value In the first step, that is, in the current training time step, the action design data is executed through the microservice architecture system , and record the execution action design data After the current experience trajectory is obtained, the current experience trajectory will be stored in the historical experience trajectory library. The historical experience trajectory library is used to store the historical experience trajectory recorded in the previous training time step of the initialization choreography model. The previous training time step here is any training time step before the current training time step T. The historical experience trajectory includes the action design data involved in the previous training time step , preference vector w, expected reward vector value And other parameters.

[0108] In the calculation function loss value In the second step, the embodiment of the present invention calculates the total benefit value in the sampling experience trajectory. When calculating the total benefit value, an experience trajectory sampling balance mechanism is adopted, that is, the historical experience trajectory and the current experience trajectory are taken into account at the same time. Specifically, the total benefit value of the microservice architecture system for the service completion delay index and the system load balancing index is determined according to the historical experience trajectory and the current experience trajectory, and then the subsequent steps are performed, that is, the value function loss value of the initialization orchestration model is constructed according to the total benefit value.

[0109] It should be noted here that the historical experience trajectory includes historical environmental state data (for example, represented as ), the current experience trajectory includes the current environment state data (e.g., represented as ), the historical target service request in the historical environment state data is the same as the current target service request in the current environment state data, that is, both can represent , but the historical preference vector in the historical environmental state data and the current preference vector in the current environment state data different.

[0110] Here, when using the sampling balance mechanism to sample the experience trajectory to calculate the loss value of the value function, not only the current experience trajectory of the current training time step T is sampled, but also the historical experience trajectory of the previous training time step is sampled, which can make the judgment network realize that even when facing the same target service request , different preference vectors w can still generate different orchestration strategies. Therefore, through the empirical trajectory sampling balance mechanism, the initialization orchestration model will focus on the active preference vector w to quickly complete the focused optimization goal. This allows the model to maintain the trained strategy and will not cause the current preference vector to overfit and forget the past orchestration strategy during the training process, thereby improving the rapid convergence of the initialization orchestration model on the new preference vector.

[0111] In step 1024, the sum of the strategy loss value and the value function loss value is used as the final loss value of the initialized orchestration model in the current training time step, and the initialized orchestration model is trained with the final loss value to obtain a microservice orchestration model.

[0112] Through the above step 1023, in the initialization orchestration model, a strategy network is constructed The strategy loss value The loss value of the value function of the critic network Afterwards, if Figure 5 As shown, the strategy loss value The loss value with the value function The sum of is used as the final loss value of the initialized orchestration model in the current training time step T , and through the final loss value Train the initialization orchestration model to obtain the microservice orchestration model.

[0113] The final loss value during training Perform self-learning according to the set learning rate , when the model is updating the parameters, the parameters Updated to , enter the next training time step T+1, and at the next training time step T+1, continue to repeat the training steps of the current training time step T to train the policy network and the judgment network at the same time. When convergence occurs, the training process is completed. The initialization encoding model after training is the microservice orchestration model.

[0114] Continue to see Figure 1 In step 103, when the microservice architecture system obtains the user service request, the request status information of the user service request is obtained.

[0115] After the microservice orchestration model is trained, it can be deployed to the microservice architecture platform, for example, it can be deployed to the terminal layer. When the microservice architecture system obtains a user service request, it obtains the request status information of the user service request, which includes the target microservice to be executed and the user preference characteristics. The target microservice is the containerized microservice that needs to be deployed when the microservice architecture system processes the user service request, and the user preference characteristics are used to characterize the user's preference for the service completion delay indicator and the system load balancing indicator.

[0116] In step 104, the request status information is input into the microservice orchestration model for policy orchestration processing to obtain an orchestration strategy for the target microservice.

[0117] After the microservice architecture system obtains the request status information of the user service request, it directly inputs the request status information into the microservice orchestration model. The microservice orchestration model immediately parses the request status information and determines the target microservice included in the user service request. The target microservice here is the containerized microservice that needs to be deployed when the microservice architecture system processes the user service request. From this, the microservice orchestration model can know the containerized microservice that needs to be deployed when processing the user service request.

[0118] In addition, the microservice orchestration model parses the user preference vector based on the user preference characteristics, and then performs policy orchestration processing for the containerized microservices and the user preference vector. Specifically, the policy network of the microservice orchestration model generates action design data based on the containerized microservices and the user preference vector, and evaluates the generated action design data through the judgment network, that is, optimizes the optimization target based on the user preference characteristics, that is, maximizes the balance of the service completion delay index and the system load balancing index after the microservice architecture module executes the action design data. When the service completion delay index and the system load balancing index reach the maximum balance, the corresponding action design data at this time is output as the orchestration strategy for the target microservice and returned to the terminal layer of the microservice architecture model.

[0119] In step 105, the containerized microservices are deployed according to the orchestration strategy, and the deployed containerized microservices are run through the microservice architecture system to process user service requests.

[0120] Through step 104, after the microservice orchestration model outputs the corresponding orchestration strategy of the containerized microservice, that is, after outputting the corresponding action design data, the terminal layer of the microservice architecture system deploys the containerized microservice according to the orchestration strategy, that is, the terminal layer deploys each containerized microservice m to be deployed to the edge node n in the edge layer for execution.

[0121] After the action design data is run, that is, after the deployment work is completed, the edge node of the microservice architecture system transmits information to the base station, and the deployed containerized microservices are respectively run to process the user service request and return to the terminal layer. The process of containerized microservice execution is also the process of user service request processing. After the containerized microservice is executed, it means that the user service request has also been processed.

[0122] Since the action design data output by the microservice orchestration model (i.e., the deployment action for the edge node) is orchestrated according to user preference characteristics, this deployment action balances the two conflicting indicators of service completion delay and system load balancing. Therefore, the optimization of the service completion delay indicator makes the process of processing user service requests not only meet user needs and improve user service experience, but also the optimization of the system load balancing indicator can improve the service performance of the microservice architecture system and reduce management costs.

[0123] Through the embodiment of the present invention, the service completion delay index and the system load balancing index of the microservice architecture system are first determined, and then the initialization orchestration model is orchestrated and trained according to these two indicators to obtain the microservice orchestration model. When there is a user request, the request status information containing the user preference characteristics is input into the microservice orchestration model, so that the microservice orchestration model will generate an orchestration strategy for the containerized microservice according to the user preference characteristics, and then the microservice architecture system can deploy the containerized microservice according to the orchestration strategy. Thus, it is achieved to use the containerized microservice orchestration model to formulate an orchestration strategy according to the user's preference for the two indicators to reasonably balance the two indicators of the service completion delay index and the system load balancing index, thereby improving the service performance of the microservice architecture system and reducing management costs.

[0124] The containerized microservices intelligent collaborative orchestration device provided by the present invention is described below. The containerized microservices intelligent collaborative orchestration device described below and the containerized microservices intelligent collaborative orchestration method described above can be referenced to each other.

[0125] Figure 6 is a schematic diagram of the structure of the containerized microservice intelligent collaborative orchestration device provided by the present invention, such as Figure 6As shown, the present invention provides a containerized microservice intelligent collaborative orchestration device, which specifically includes: a determination module 601, a training module 602, an acquisition module 603, an orchestration module 604, and a deployment module 605. Among them, the determination module 601 is used to determine the service completion delay index of the microservice architecture system, and determine the system load balancing index of the microservice architecture system based on the environmental information of the microservice architecture system. The service completion delay index is the time required for the microservice architecture system to process a user service request, and the system load balancing index is used to characterize the degree of system load imbalance of the microservice architecture system when processing user service requests; the training module 602 is used to perform orchestration training on the initialization orchestration model based on the service completion delay index and the system load balancing index to obtain a microservice orchestration model; the acquisition module 603 is used when the microservice architecture system obtains a user service request, Obtain request status information of the user service request, wherein the request status information includes a target microservice and user preference characteristics, the target microservice is a containerized microservice that needs to be deployed when the microservice architecture system processes the user service request, and the user preference characteristics are used to characterize the user's preference for the service completion delay indicator and the system load balancing indicator; an orchestration module 604 is used to input the request status information into the microservice orchestration model for policy orchestration processing to obtain an orchestration strategy for the target microservice; a deployment module 605 is used to deploy the containerized microservice according to the orchestration strategy, and run the deployed containerized microservice through the microservice architecture system to process the user service request.

[0126] Specifically, the above-mentioned containerized microservice intelligent collaborative orchestration device provided by the present invention can implement all the method steps implemented by the above-mentioned containerized microservice intelligent collaborative orchestration method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be specifically repeated here.

[0127] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communications interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the containerized microservice intelligent collaborative orchestration method, the method comprising: determining the service completion delay index of the microservice architecture system, and determining the system load balancing index of the microservice architecture system according to the environmental information of the microservice architecture system, the service completion delay index is the time required for the microservice architecture system to process a user service request, and the system load balancing index is used to characterize the degree of system load imbalance of the microservice architecture system when processing a user service request; based on the service completion delay index and the system load balancing index, the initialization orchestration model is orchestrated and trained to obtain a microservice orchestration model; when the microservice When the service architecture system obtains a user service request, it obtains request status information of the user service request, wherein the request status information includes a target microservice and a user preference feature, the target microservice is a containerized microservice that needs to be deployed when the microservice architecture system processes the user service request, and the user preference feature is used to characterize the user's preference for the service completion delay indicator and the system load balancing indicator; the request status information is input into the microservice orchestration model for policy orchestration processing to obtain an orchestration strategy for the target microservice; the containerized microservice is deployed according to the orchestration strategy, and the deployed containerized microservice is run through the microservice architecture system to process the user service request.

[0128] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0129] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the containerized microservice intelligent collaborative orchestration method provided by the above methods, the method including: determining a service completion delay index of the microservice architecture system, and determining a system load balancing index of the microservice architecture system based on environmental information of the microservice architecture system, the service completion delay index being the time required for the microservice architecture system to process a user service request, and the system load balancing index being used to characterize the degree of system load imbalance of the microservice architecture system when processing a user service request; based on the service completion delay index and the system load balancing index , perform orchestration training on the initialization orchestration model to obtain a microservice orchestration model; when the microservice architecture system obtains a user service request, obtain request status information of the user service request, wherein the request status information includes a target microservice and a user preference feature, the target microservice is a containerized microservice to be deployed when the microservice architecture system processes the user service request, and the user preference feature is used to characterize the user's preference for the service completion delay indicator and the system load balancing indicator; input the request status information into the microservice orchestration model for strategy orchestration processing to obtain an orchestration strategy for the target microservice; deploy the containerized microservice according to the orchestration strategy, and run the deployed containerized microservice through the microservice architecture system to process the user service request.

[0130] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the containerized microservice intelligent collaborative orchestration method provided by the above-mentioned methods, the method comprising: determining a service completion delay index of the microservice architecture system, and determining a system load balancing index of the microservice architecture system based on environmental information of the microservice architecture system, the service completion delay index being the time required for the microservice architecture system to process a user service request, and the system load balancing index being used to characterize the degree of system load imbalance of the microservice architecture system when processing a user service request; based on the service completion delay index and the system load balancing index, compiling the initialization orchestration model; The microservices architecture system performs training to obtain a microservices orchestration model; when the microservices architecture system obtains a user service request, the request status information of the user service request is obtained, wherein the request status information includes a target microservice and a user preference feature, the target microservice is a containerized microservice to be deployed when the microservices architecture system processes the user service request, and the user preference feature is used to characterize the user's preference for the service completion delay indicator and the system load balancing indicator; the request status information is input into the microservices orchestration model for strategy orchestration processing to obtain an orchestration strategy for the target microservice; the containerized microservice is deployed according to the orchestration strategy, and the deployed containerized microservice is run through the microservices architecture system to process the user service request.

[0131] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A containerized microservice intelligent collaborative orchestration method, applied to a microservice architecture system, characterized in that: The method comprises: Determine a service completion delay index of the microservice architecture system, and determine a system load balancing index of the microservice architecture system based on the environmental information of the microservice architecture system, wherein the service completion delay index is the time required for the microservice architecture system to process a user service request, and the system load balancing index is used to characterize the degree of system load imbalance of the microservice architecture system when processing a user service request; Based on the service completion delay indicator and the system load balancing indicator, the initialization orchestration model is orchestrated and trained to obtain a microservice orchestration model; When the microservice architecture system obtains a user service request, the request status information of the user service request is obtained, wherein the request status information includes a target microservice and a user preference feature, the target microservice is a containerized microservice to be deployed when the microservice architecture system processes the user service request, and the user preference feature is used to characterize the user's preference for the service completion delay indicator and the system load balancing indicator; Inputting the request status information into the microservice orchestration model for strategy orchestration processing to obtain an orchestration strategy for the target microservice; The containerized microservice is deployed according to the orchestration strategy, and the deployed containerized microservice is run through the microservice architecture system to process the user service request.

2. According to claim 1, a containerized microservice intelligent collaborative orchestration method is characterized in that: The microservice architecture system includes a terminal layer, an edge layer, and an image layer. The terminal layer includes multiple containerized microservices obtained by splitting user service requests. The edge layer includes multiple edge nodes. The image layer includes multiple container images. The container images are used to deploy the containerized microservices to the edge nodes for execution. The service completion delay index of the microservice architecture system is determined, including: When the containerized microservice is deployed to the edge node, in the edge node, determining an image pulling time for the containerized microservice to pull a container image from the image layer; After the containerized microservice pulls the container image, determining the service processing time of the edge node processing the containerized microservice, and determining the data transmission time between the edge nodes; A service completion delay index of the microservice architecture system is determined according to the image pulling time, the service processing time, and the data transmission time.

3. According to claim 1, a containerized microservice intelligent collaborative orchestration method is characterized in that: The microservice system architecture includes an edge layer, the edge layer includes multiple edge nodes, the environmental information includes remaining computing resources and remaining memory of each edge node, and determining the system load balancing index of the microservice architecture system according to the environmental information of the microservice architecture system includes: For each of the edge nodes, determine a load computing resource according to the remaining computing resource, and determine a load memory according to the remaining memory; Determining a first average value of a load computing resource of the edge node, and determining a second average value of a load memory of the edge node; The total load balancing value of the edge nodes in the microservice architecture system is determined according to the first average value and the second average value, and the average of the total load balancing values ​​is used as the system load balancing indicator of the microservice architecture system.

4. According to claim 1, a containerized microservice intelligent collaborative orchestration method is characterized in that: The initialization orchestration model includes a policy network and a judgment network. The training process of the initialization orchestration model is performed according to the training time step. The initialization orchestration model is orchestrated and trained based on the service completion delay index and the system load balancing index to obtain a microservice orchestration model, including: Acquire environmental status data, wherein the environmental status data includes a target service request and a preference vector, wherein the preference vector is constructed based on a delay weight and a load weight, wherein the delay weight represents a preference of the target service request for the service completion delay indicator, and the load weight represents a preference of the target service request for the system load balancing indicator; At the current training time step, the environment state data is input into the policy network to generate action design data, and the evaluation network is called to predict the environment state data to generate an expected reward vector value for the action design data, wherein the action design data is used to characterize the edge node deployment action, and the edge node deployment action is a deployment action of deploying the containerized microservice of the terminal layer to the edge node in the edge layer when the microservice architecture system processes the target service request, and the current time step is any training time step in the orchestration training process; Constructing a strategy loss value of the initialized orchestration model according to the action design data, and constructing a value function loss value of the initialized orchestration model according to the action design data and the expected reward vector value; The sum of the strategy loss value and the value function loss value is used as the final loss value of the initialized orchestration model in the current training time step, and the initialized orchestration model is trained by the final loss value to obtain a microservice orchestration model.

5. A containerized microservice intelligent collaborative orchestration method according to claim 4, characterized in that: The step of constructing the strategy loss value of the initialization orchestration model according to the action design data includes: Obtaining the prediction probability of the policy network when generating action design data at the current training time step, and determining the policy amplitude value of the policy network between training time steps according to the prediction probability; In the current training time step, the generalized advantage estimation values ​​of the service completion delay index and the system load balancing index are determined respectively; Determine, according to the generalized advantage estimate and the preference vector included in the environmental state data, a delay advantage value of the policy network of the initialization orchestration model for the service completion delay indicator, and a load advantage value of the policy network for the system load balancing indicator; Determining an advantage function value of the strategy network according to the delay advantage value and the load advantage value; A strategy loss value of the initialized orchestration model is constructed according to the advantage function value and the strategy amplitude value.

6. A containerized microservice intelligent collaborative orchestration method according to claim 4, characterized in that: Constructing the value function loss value of the initialized orchestration model according to the action design data and the expected reward vector value, including: In the current training time step, the action design data is executed through the microservice architecture system, and the current experience trajectory of executing the action design data is recorded, wherein the current experience trajectory includes the action design data, the preference vector, and the expected reward vector value; Determine the total revenue value of the microservice architecture system for the service completion delay index and the system load balancing index according to the current experience trajectory, and determine the discounted revenue value of the total revenue value; Determine a cumulative reward vector value in a current training time step, and determine an average value of a difference between the cumulative reward vector value and the discounted benefit value; A value function loss value of the initialized orchestration model is constructed based on the average value of the difference and the preference vector.

7. A containerized microservice intelligent collaborative orchestration method according to claim 6, characterized in that: Determining the total benefit value of the microservice architecture system for the service completion delay index and the system load balancing index according to the current experience trajectory includes: Obtain the reward vector value of the service completion delay indicator and the system load balancing indicator; Determining a linear reward scalar value according to the preference vector included in the current experience trajectory and the reward vector value; Determine a penalty value obtained by the microservice architecture system after executing the constraint deployment action when executing the action design data corresponding to the current experience trajectory; The difference between the linear reward scalar value and the penalty value is used as the total benefit value of the microservice architecture system for the service completion delay index and the system load balancing index.

8. A containerized microservice intelligent collaborative orchestration method according to claim 6, characterized in that: After recording the experience track of executing the action design data, the method further includes: Storing the current experience trajectory in a historical experience trajectory library, wherein the historical experience trajectory library is used to store the historical experience trajectory recorded by the initialized orchestration model at a previous training time step, and the previous training time step is any training time step before the current training time step; Determine the total benefit value of the microservice architecture system for the service completion delay index and the system load balancing index based on the historical experience trajectory and the current experience trajectory, and construct the value function loss value of the initialization orchestration model based on the total benefit value; Among them, the historical experience trajectory includes historical environment state data, the current experience trajectory includes current environment state data, the historical target service request in the historical environment state data is the same as the current target service request in the current environment state data, and the historical preference vector in the historical environment state data is different from the current preference vector in the current environment state data.

9. A containerized microservice intelligent collaborative orchestration device, characterized in that: The device comprises: A determination module is used to determine a service completion delay index of the microservice architecture system, and determine a system load balancing index of the microservice architecture system according to environmental information of the microservice architecture system, wherein the service completion delay index is the time required for the microservice architecture system to process a user service request, and the system load balancing index is used to characterize the degree of system load imbalance of the microservice architecture system when processing a user service request; A training module, used to perform orchestration training on the initialization orchestration model based on the service completion delay indicator and the system load balancing indicator to obtain a microservice orchestration model; An acquisition module, used for acquiring request status information of the user service request when the microservice architecture system acquires the user service request, wherein the request status information includes a target microservice and a user preference feature, the target microservice is a containerized microservice to be deployed when the microservice architecture system processes the user service request, and the user preference feature is used to characterize the user's preference for the service completion delay indicator and the system load balancing indicator; An orchestration module, used to input the request status information into the microservice orchestration model for strategy orchestration processing to obtain an orchestration strategy for the target microservice; A deployment module deploys the containerized microservice according to the orchestration strategy, and runs the deployed containerized microservice through a microservice architecture system to process the user service request.

Images