Method and system for microservice deployment and orchestration optimization of deep neural networks

Abstract

The application provides a deep neural network micro-service deployment and arrangement optimization method and system, which comprises the following steps: deploying multiple micro-services composed of different container images to multiple edge server nodes based on micro-service deployment startup time and resource conditions by a cloud center node, wherein each micro-service is constructed based on a business branch of a pre-trained multi-branch deep neural network model, and each business branch corresponds to a micro-service set; determining a deployment and arrangement joint optimization target based on micro-service deployment startup time, business time delay and business accuracy, performing joint iterative optimization including micro-service deployment optimization and micro-service arrangement optimization according to the deployment and arrangement joint optimization target, and further performing arrangement iterative adjustment based on multi-branch deep neural network model branch optimization when the result of the joint iterative optimization cannot achieve the deployment and arrangement joint optimization target, so as to achieve the deployment and arrangement joint optimization target.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things technology, and in particular to a method and system for microservice deployment and orchestration optimization of a deep neural network. Background Art

[0002] The rapid development of the Internet of Things (IoT) has driven the generation and processing of massive amounts of data. In-depth research in big data and artificial intelligence (AI), along with the implementation of efficient computing technologies, has become crucial to addressing this data.

[0003] Advances in deep learning have significantly boosted the development and application of AI, achieving human-level accuracy in fields such as computer vision and speech recognition. The weights of neurons in deep neural networks (DNNs) are optimized through data training to meet specific application requirements and tasks. However, this high accuracy comes with enormous computational and memory requirements. To address resource constraints, researchers are leveraging the deep structure of DNNs and combining accuracy requirements with computational cost to compress models. Research has shown that neural networks with different structures can be used for inference, resulting in varying computational loads and accuracy. Using shallow neural networks can capture the most information, effectively reducing computational load and latency while maintaining accuracy requirements.

[0004] Edge computing brings computing and data storage closer to where it's needed, avoiding network bottlenecks in central cloud computing locations. This is particularly important for emerging real-time AI applications, particularly those driven by deep neural networks in intelligent environments that require immediate decision-making. Furthermore, distributing the computing load across multiple edge server nodes improves network scalability through horizontal scaling, resource optimization, and geographic distribution. These advantages make edge computing a vital component of modern IT infrastructure, capable of supporting a wide range of applications with high performance and reliability.

[0005] However, since deep learning training and inference consume a lot of computing resources, the effectiveness of its calculations directly affects the real-time and effectiveness of artificial intelligence applications. How to combine the current real-time and scalable edge computing to improve the real-time and effectiveness of deep neural network inference applications under limited edge resource conditions is an urgent problem that needs to be solved. Summary of the Invention

[0006] In view of this, an embodiment of the present invention provides a microservice deployment and orchestration optimization method for a deep neural network to eliminate or improve one or more defects in the prior art.

[0007] One aspect of the present invention provides a method for microservice deployment and orchestration optimization of a deep neural network. The method is based on a distributed architecture comprising a cloud center node and edge server nodes in communication with the cloud center node. The method comprises:

[0008] Initial deployment step: The cloud center node deploys multiple microservices composed of different container images to multiple edge server nodes based on the microservice deployment startup time and resource conditions. Each microservice is built based on the business branch of the pre-trained multi-branch deep neural network model, and each business branch corresponds to a set of microservices.

[0009] Deployment and orchestration joint optimization step: Determine the deployment and orchestration joint optimization target based on the microservice deployment startup time, business latency and business accuracy, and perform joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the deployment and orchestration joint optimization target. When the result of the joint iterative optimization cannot achieve the deployment and orchestration joint optimization target, further perform orchestration iterative adjustment based on multi-branch deep neural network model branch optimization to achieve the deployment and orchestration joint optimization target.

[0010] In some embodiments of the present invention, the microservice flow of each business branch is constructed based on a directed acyclic graph; the branch optimization of the multi-branch deep neural network model is a branch optimization algorithm based on a branch early exit algorithm.

[0011] In some embodiments of the present invention, the deployment of multiple microservices consisting of different container images to multiple edge server nodes based on the microservice deployment startup time and resource conditions includes: pulling the container image corresponding to each microservice from the image layer set in the image repository, downloading the pulled container image to the matching edge server according to pre-set constraints, thereby deploying multiple microservices consisting of different container images to multiple edge server nodes; the constraints include: the startup time of the deployed microservice is less than the predetermined image download delay threshold; the remaining computing resources and storage resources of the edge server node where the microservice is to be deployed are greater than the computing resources and storage resources required by the microservice.

[0012] In some embodiments of the present invention, the microservice deployment optimization includes microservice deployment reduction; and the microservice orchestration optimization includes replacing edge server nodes of the microservice deployment.

[0013] In some embodiments of the present invention, the deployment and orchestration joint optimization objective is determined based on the following formula:

[0014]

[0015] Where F(ψ) represents the joint optimization objective of deployment and orchestration, ψ is the optimization variable, N represents the number of edge server nodes, U represents the number of branches of the deep neural network model, I represents the number of microservices, ω1 represents the service delay weight parameter, ω2 represents the service accuracy weight parameter, ω3 represents the deployment delay weight parameter, and j(n) represents the service reached by edge server node n; τ j(n) Indicates the download start time of service j(n), The value of indicates whether to select the branch u of the deep neural network model to process business j(n), e u Represents the computational accuracy of branch u of the deep neural network model; Indicates the download and startup time required for deploying microservice i on edge server node n.

[0016] The optimization variables are: Among them, {b i，n} represents the deployment decision of the microservice on the edge server node (deployment variable), represents the branch selection decision of the deep neural network model (branch selection variable), represents the microservice orchestration decision (orchestration variable), The value of indicates whether microservice i on edge server node n′ processes business j(n).

[0017] In some embodiments of the present invention, performing joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the deployment and orchestration joint optimization goal includes:

[0018] Calculate the gradient of the deployment and orchestration joint optimization objective function for the deployment variables of the current microservice;

[0019] Based on the calculated gradient value and the number of edge server nodes where the current microservice is deployed, a relative gradient value between deploying the current microservice on one edge server node and the average deployment on other edge server nodes is calculated, and a microservice deployment reduction strategy is determined by selecting the deployment with the largest relative gradient value among all deployments.

[0020] The joint iterative optimization includes: orchestrating the microservices on the microservice flow of the business to the edge server node deployment with the minimum transmission and computing cost through orchestration update based on the microservice deployment reduction strategy;

[0021] When the result of the joint iterative optimization cannot achieve the deployment and orchestration joint optimization goal, further orchestration iterative adjustment based on the multi-branch deep neural network model branch optimization is performed, including: when the optimization target value of the joint iterative optimization no longer decreases and meets the branch optimization constraint conditions, triggering the multi-branch deep neural network model branch iterative optimization, and performing orchestration update after each branch optimization, so as to orchestrate the microservices on the business's microservice flow to the edge server node deployment with the lowest transmission and computing cost through orchestration update, until the deployment and orchestration joint optimization goal is achieved.

[0022] In some embodiments of the present invention, the step of calculating the gradient value of the deployment and orchestration joint optimization objective function for the deployment variables of the current microservice includes: calculating the gradient value of the deployment and orchestration joint optimization objective function for the deployment variables of each edge server node where the current microservice is deployed, and calculating the total gradient value of the deployment and orchestration joint optimization objective function for the deployment variables of all edge server nodes where the current microservice is deployed based on the calculated gradient values;

[0023] The method of orchestrating the microservices on the microservice flow of the business to the edge server node deployment with the minimum transmission and computing cost through orchestration update includes: calculating the gradient value of the corresponding deployment and orchestration joint optimization target for the microservice orchestration variable based on the orchestration update decision, selecting the orchestration update decision corresponding to the minimum gradient value to obtain the edge server node with the minimum transmission and computing cost, and orchestrating the microservices on the microservice flow of the business to the selected server node.

[0024] In some embodiments of the present invention, in the initial deployment step, the front and back microservices on the microservice flow are deployed on the nearest edge server node, and the longest branch in the multi-branch deep neural network model is used as the currently selected branch.

[0025] The above solution introduces container-based microservices, combines the branch structure of deep neural networks with decomposable microservices, and uses multiple independent microservices to complete complex deep neural network inference calculations. By optimizing the deployment of microservices on multiple edge computing platforms and completing the microservice orchestration of corresponding tasks, the real-time performance and effectiveness of calculations can be improved under the reasonable allocation and scheduling of limited resources.

[0026] The second aspect of the present invention also provides a microservice deployment and orchestration optimization system for a deep neural network, which includes a computer device, wherein the computer device includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0027] The third aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned method for microservice deployment and orchestration optimization of deep neural networks.

[0028] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0029] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0031] Figure 1 Schematic diagram of a method for optimizing microservice deployment and orchestration of a deep neural network in accordance with an embodiment of the present invention.

[0032] Figure 2 Schematic diagram of collaborative microservice deployment and orchestration in one embodiment of the present invention.

[0033] Figure 3 A schematic diagram of the relationship between the CPU occupancy rate and computing time of an edge server node in an embodiment of the invention.

[0034] Figure 4 Schematic diagram of a multi-branch DNN structure in one embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0036] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0037] In order to flexibly, quickly, and efficiently deploy AI applications at the edge of the network and achieve the objectives of the present invention, the present invention adopts container virtualization technology to propose a microservice (MS) deployment and orchestration optimization method for deep neural networks. Container virtualization technology enables an independent container engine, encapsulating the application and its dependencies in a container, allowing the application and its dependencies to run in an isolated environment. In the architecture deployment based on container virtualization, microservices, as a key technology, have the advantage of loose coupling. By deploying container-based microservices on multiple edge server nodes, the deployment architecture of the present invention can decompose complex applications into smaller, interconnected components, each running its own microservice. By encapsulating each service in its own container, microservices can be independently developed, deployed, and scaled. In addition, when there are overlapping microservices in an application, components can be shared, and each component can be updated independently without affecting other components. Therefore, container-based microservice deployment can not only meet the latency requirements of resource and service deployment in edge computing, but also enhance the elasticity and scalability of services.

[0038] The method of the present invention is based on a distributed architecture, such as Figure 2 As shown, the distributed architecture may include a cloud center server node (hereinafter referred to as cloud center node) and a number of edge server nodes (or edge nodes, EN for short) (e.g., N, N = {n, n = 1, ...n, ..., N}) that are connected to the cloud center node. The resource capacity of the cloud center node is stronger than that of the edge server node. The cloud center node deploys I container-based microservices (microservice set I = {i, i = 1, ..., I}) on each edge server node based on deep neural networks, edge computing, and container-based microservice technologies. The i-th microservice is in Figure 2 Indicated as MSi. Figure 2 As shown, the microservice MSi is initially deployed on the nth edge server node (ENn) based on the microservice deployment startup time (image pull delay, or download delay) and the resource conditions of the edge server node. Due to the different business arrival volumes of each edge server, it is possible that the queuing time of tasks in the service queues of some edge server nodes is too long. At this time, the deployment and orchestration can be jointly optimized based on the task queuing delay of the microservice at the edge server node and the data transmission delay of the microservice flow between the edge server nodes. In addition, in the joint optimization process of deployment and orchestration, the early exit mechanism of the deep neural network branch can be further combined to improve the optimization, that is, the DNN branch is selected for deployment and orchestration, and the optimization choice is made between the system resource executable and accuracy requirements. Therefore, through the collaboration between edge servers, task requests from users can be processed quickly and efficiently, and the corresponding deep neural network reasoning can be completed. Figure 2In the figure, i, i′, and i″ have the same meaning and all represent microservices. Different symbols are used only to distinguish them and represent different microservices. Similarly, n, n′, and n″ also have the same meaning and all represent edge server nodes. Different symbols are used only to distinguish them and represent different edge server nodes.

[0039] like Figure 1 As shown, the microservice deployment and orchestration optimization method of the deep neural network of the present invention includes a microservice initial deployment step S110 and a deployment and orchestration joint optimization step S120.

[0040] In the initial deployment step S110, the cloud center node deploys multiple microservices composed of different container images to multiple edge server nodes based on the microservice deployment startup time and resource conditions.

[0041] The cloud center node is built with a pre-trained multi-branch DNN model, and all container images are deployed in the image warehouse of the cloud center node. All images in the warehouse can be equivalent to a multi-layer (such as L layers) image structure, that is, composed of multiple image layers, and the image layer set can be expressed as L = {1, ..., L}. To deploy container-based microservices on edge server nodes, it is first necessary to pull the container image from the image warehouse of the cloud center node and download it to the edge server, so as to deploy multiple microservices composed of different container images to multiple edge server nodes, thereby using the downloaded image to start one or more containers. In an embodiment of the present invention, in order to solve the problem of real-time and effectiveness of deep neural network inference applications, each microservice is constructed based on a business branch of a pre-trained multi-branch deep neural network (DNN) model, and each business branch corresponds to a set of microservices.

[0042] During microservice deployment, it's important to consider the microservice startup time and resource requirements of each edge server node. The microservice startup time can include the microservice download startup time (also known as image download latency or pull latency), and resource requirements can include computing resources and storage resources (such as memory resources and / or cache resources). In other words, in addition to considering the microservice startup time, it's also necessary to consider whether the edge server node can support the operation of the microservice.

[0043] Assume that all images are pre-deployed on the cloud center node, and some image layers are pre-cached on an edge server node. If you want to deploy microservices to the edge server node, you can use β l，i Indicates whether the edge server node is required to download the image layer for the microservice, such as β l，i =1 means that the image layer l required by microservice i is not available locally on the edge server node, so it needs to be downloaded; β l，i= 0 means that the image layer l required by microservice i is not needed or has been cached at the edge server node, so it does not need to be downloaded. i，n =1 means that edge server node n needs to deploy microservice i, then b i，n = 0 means that edge server node n does not need to deploy microservice i. In this case, when deploying microservices composed of multiple image layers to different server nodes, the image download delay required is It can be expressed as:

[0044]

[0045] Among them, b i，n Used to identify whether server n deploys microservice i; β l，i It is used to identify whether it is necessary to download image layer 1 for microservice i. s1 represents the file size of image layer l, that is, the memory boundary of the lth layer image data of the microservice. n Indicates the download rate. n It can be calculated based on the bandwidth and signal-to-noise ratio from the cloud center node to the edge server node n.

[0046] based on The download startup time of global deployment can also be obtained. The download startup time of global deployment can represent the sum of the download startup time required for deployment of all deployed microservices on all edge server nodes. For example, the download startup time of global deployment can be expressed as in, Indicates the download and startup time required for deploying microservice i on edge server node n.

[0047] From this we can see that when deploying microservices i with different image structures to different nodes n, there will be different image download delays. When deploying microservices, to ensure microservice startup efficiency, the startup time (image download latency) of deploying microservices on edge server nodes can be set to be less than a predetermined time threshold. In the initial deployment step of the present invention, microservices are preferably orchestrated to edge server nodes with low transmission and computational costs. For example, the preceding and following microservices in a microservice flow are deployed on the nearest edge server node.

[0048] In addition, suppose that the resource configuration of microservice i requires a CPU usage of f i , the storage resource requirement is s′ i , the total load of computing resources and storage resources of the microservices deployed on the edge server node cannot exceed the total amount of computing resources and storage resources (such as memory resources) of the corresponding node. That is, it must meet the following requirements:

[0049]

[0050] Among them, s′ i =∑ l β l，i s l , F n and S n They represent the available computing resource boundary and memory resource boundary on server node n respectively.

[0051] Furthermore, when deploying microservices i with different image structures to different nodes n, the remaining computing resources and storage resources of the edge server node where the microservice is to be deployed should be greater than the computing resources and storage resources required by the microservice.

[0052] Therefore, in an embodiment of the present invention, multiple microservices composed of different container images are deployed to multiple edge server nodes based on the microservice deployment startup time and resource conditions. Specifically, this may include: pulling the container image corresponding to each microservice from the image layer set in the image repository, downloading the pulled container image to the matching edge server according to pre-set constraints, thereby deploying multiple microservices composed of different container images to multiple edge server nodes; wherein, the constraints may include: the startup time of the deployed microservice is less than a predetermined time threshold, and the remaining computing resources and storage resources of the edge server node where the microservice is to be deployed are greater than the computing resources and storage resources required by the microservice. This constraint is only an example, and the present invention is not limited to this.

[0053] The fine-grained structure of microservices allows for dynamic adjustment of infrastructure based on demand for different types of services. For example, clusters for compute-intensive services can be interleaved with clusters for I / O-intensive services, improving both computing resource utilization and optimizing network bandwidth usage.

[0054] In addition, it is also important to note that under the condition that the total amount of resources is not exceeded, the resource load still has a significant impact on system performance, especially service execution performance. Here is a parallel I / O benchmark test in a multi-threaded environment. The verification results show that as the CPU usage increases, the calculation time gradually increases, such as Figure 3 As shown, 10 5 Taking digital computing as an example, when the CPU utilization rate is 80%, the computing time is six times that when the utilization rate is 20%. It can be seen that the resource load rate affects the processing speed of the edge server node.

[0055] Therefore, in a preferred embodiment of the present invention, when deploying microservices, the computing resource occupancy rate (resource load rate) can be further considered to set or optimize constraints to maximize the reasoning efficiency.

[0056] Deep neural network application services are very different from other services in terms of logical behavior, system resource usage, etc. A single edge server node with limited resources is difficult to meet the computational timeliness of deep neural network training involved in the business. When the deep neural network microservices are deployed to multiple edge server nodes as mentioned above, the real-time and effectiveness of deep neural network reasoning applications can be greatly improved. However, due to the real-time nature of the application, the edge server nodes may still have problems with insufficient computing resources or poor reasoning efficiency. The multi-branch DNN network structure can select networks of different structures for reasoning based on the early exit mechanism of branches. The computational amount and actuarial accuracy are different. If a partially shallow network is used, most of the information can still be captured, and the computational amount and computational delay can be effectively reduced at the cost of minimal accuracy loss. Therefore, in the embodiment of the present application, the constructed DNN network structure is a pre-trained multi-branch DNN network structure, such as Figure 4 As shown (for example only), it is possible to dynamically adjust computing requirements based on the computational complexity of the input data and the business accuracy requirements, and save computing power and improve inference efficiency by selectively activating its internal modules (selecting appropriate branches), thereby meeting different service accuracy and latency requirements. However, how to consider microservice deployment, orchestration optimization, and DNN branch structure for further optimization is a difficult problem to solve due to the strong coupling relationship between microservice deployment, DNN branch selection, and orchestration. This is also a problem that this invention will further solve.

[0057] During the microservice deployment phase of the present invention, each microservice is built based on the business branches of a pre-trained multi-branch deep neural network model, with each business branch corresponding to a microservice set. More specifically, based on the container architecture, the microservice flows of different DNN business branches can be constructed based on a directed acyclic graph (DAG).

[0058] More specifically, the u-th branch of a specific DNN business is represented by the graph G u =(I u , ε u ) indicates that u=1,…,U, where U is the number of branches of the business. The microservice set of the u-th branch is I u , As a node of DAG, the matrix It is the microservice association relationship on the u-th branch of the business, that is, the set of edges in the DAG, and its expanded form is:

[0059]

[0060] Among them, I u =|I u |, ε u，v，v′ ∈{0, 1} is ε uThe element in the vth row and v′ column of the business is used to identify the uth branch based on the microservice set I u Microservice association relationship; v, v′=1,…,I u ;i u，v Indicates I u The vth element of . When δ u，v，v′ =1 indicates that there is a dependency relationship between microservices, that is, microservice i u，v ∈I u It is a microservice u，v′ ∈I u Pre-order operation, microservice i u，v The output of microservice i u，v′ Input; when ε u，v，v′ = 0, microservice i u，v and microservices u，v′ There is no dependency between them.

[0061] The pre-trained multi-branch DNN model has a branch early exit mechanism for different businesses. It obtains business accuracy and corresponding computing requirements, and selects different inference accuracy e according to different computing requirements, latency requirements, etc. u (mean square loss) branch u, where u≤U and satisfies |I u |<|I u+1 |≤I,e u >e u+1 Preferably, in the initial deployment step, when performing branch selection, the longest branch among the DNN branches is preferentially selected as the current branch by default. In the subsequent optimization process, branch selection can be performed based on the early branch withdrawal mechanism.

[0062] In order to facilitate the microservice collection of business branches I u The mapping between the complete set of microservices I is given on the basis of I = {i, i = 1, ..., I} u and

[0063] I u =ξ u I (4)

[0064] Among them, ξ u For I u and the mapping relationship matrix of I, ξ u，v，i Yes u The element in row v and column i of u，v =i, ξ u，v，i =1, otherwise ξ u，v，i = 0. Let ξ u，i for ξ u The i-th column of u，i =0, indicating that microservice i does not exist in Iu At this time, you can identify the quantity Indicates the mapping relationship value between branch u and microservice i, Indicates that microservice i does not exist in branch u, otherwise

[0065] Based on I u The mapping relationship between I and G u =(I u , ε u ), you can get the extended graph parameters in, yes The element in the i-th row and i′th column of is used to represent the microservice association relationship of the u-th branch of the business based on the microservice set I; if and only if ε u，v，v′ =1, and i=i u，v , i′=i u，v′ hour, Indicates that microservices i and i′ in the u-th branch of the business are tandem; otherwise

[0066] Through the above microservice deployment process, after the microservices corresponding to the business branches of the pre-trained multi-branch deep neural network model are deployed on multiple edge server nodes, the deep neural network model can be efficiently applied for inference calculations.

[0067] However, because AI applications are real-time, sudden surges in business volume can easily lead to resource shortages or decreased computing efficiency on edge server nodes. In such cases, multi-branch DNN inference tasks require a coordinated microservice orchestration mechanism to meet business accuracy and latency requirements. In other words, microservice orchestration optimization must be performed simultaneously with deployment to further improve the real-time performance and effectiveness of deep neural network inference applications.

[0068] Therefore, the microservice deployment and orchestration optimization method for a deep neural network of the present invention further includes a joint optimization step S120 of deployment and orchestration based on the initial deployment result of step S110. The implementation of the joint optimization of deployment and orchestration of the present invention will be described below.

[0069] Factors influencing the real-time and effectiveness of deep neural network inference applications include microservice deployment startup time, service latency, and service accuracy. Microservice deployment startup time includes image download latency, while service latency can include microservice execution latency, transmission latency, and service queuing latency, impacting the efficiency of microservices tailored to business needs. Service accuracy is related to the branch selection within the multi-branch DNN architecture, with different service accuracies corresponding to different computational loads. Multi-branch DNNs achieve varying inference accuracy and computational requirements due to different early retirement mechanisms.

[0070] In an embodiment of the present invention, a joint optimization target for deployment and orchestration can be determined based on the microservice deployment startup time, service latency, and service accuracy. Based on this determined joint optimization target, joint optimization of microservice deployment and orchestration for the entire system can be performed. The purpose of this joint optimization is to minimize the total latency caused by microservice deployment startup and service execution while ensuring service accuracy, thereby improving the efficiency of microservices oriented to service needs.

[0071] Let's analyze the factors that affect the efficiency of microservices oriented to business needs.

[0072] The arrival time and volume of services within the coverage area of ​​each edge server node are different. Considering the recent access principle of the network, the traffic volume of service j(n) arriving at edge server node n is modeled as obeying the arrival rate λ j (n) is a Poisson distribution, and the business size (computing requirements) follows a mean of μ -1 The exponential distribution of , that is, the business is modeled as conforming to the M / M / 1 model. Here, it is assumed that the business is generated on the edge side. The cloud center node does not have the initial arrival business, but can collaborate to process other edge businesses.

[0073] Taking the single instance model as an example, for the service j(n) arriving at the edge server node n, if The value of indicates whether to select the u-th branch of the multi-branch DNN to process the business j(n). When the u-th branch is selected, You can use the extended graph parameters of the microservice flow Use the microservice flow to complete the business j(n) in sequence. The value of indicates whether business j(n) is processed by microservice i on server node n′. When it is completed by microservice i orchestrated on server node n′ (that is, the corresponding microservice is deleted from the original edge server node and orchestrated to other better edge server nodes), otherwise

[0074] For the microservice flow structure of business j(n), consider the two microservice switching processes, that is, switching from the previous microservice i′ (deployed on the edge server node n′) to the microservice i″ (deployed on the edge server node n″), i″≠i′ and When microservice i″ and microservice i″ are deployed at node n′ (n′=n″) and run efficiently, there is and That is, when running microservice i″, no data transmission time is required, that is, the transmission delay when running microservice i″ Otherwise, when the edge server node n′ does not deploy microservice i″ or it has deployed but the operation burden is heavy, microservice i′ needs to be transferred to node n″ with microservice i″ for service after node n′ is completed (n″≠n′). At this time, and Taking microservice i″ as the general observation object, the average transmission time of the data to be processed for:

[0075]

[0076] Among them, r n′，n″ Indicates the transmission rate from edge server node n′ to edge server node n″. The value of is used to indicate whether server nodes n′ and n″ are the same node, if and only if n′≠n″, otherwise That is, there is no transmission delay when running on the same node. represents the unit processing computation of microservice i′, Indicates the output data volume of microservice i′. and Definition and The same, but with different subscripts to indicate microservice switching and possible node switching; j(n) represents the business volume of business j(n); Indicates whether microservices i′ and i″ in the u-th branch of the business are related. i′，n′ and b i″，n′ Definition and b i，n Consistent, indicating whether servers n, n′, and n″ deploy microservices i, i′, and i″ respectively.

[0077] Considering the microservice queue on node n″, using the first-come, first-served (FIFO) principle, the average stay time of microservice i″ in the queue on node n″ is It can be expressed as:

[0078]

[0079] in, is the average queue ratio of microservice i on node n″, is the average service time of microservice i on node n. Based on the modeling characteristics of DNN service j(n), the service model of each microservice is an M / M / 1 model. Considering the selection and unloading of microservices between nodes, and based on its queue characteristics and statistical characteristics, the average stay time of microservice i in the queue on node n is It can be expressed as:

[0080]

[0081] in:

[0082]

[0083] in, is the service arrival rate of microservice i″ on edge server node n″; n″ is the processing speed of edge server node n″; is the service calculation requirement parameter.

[0084] Then the queuing time for service j(n) to perform microservice i″ is:

[0085]

[0086] The total delay for business j(n) to complete its reasoning task needs to integrate the microservice transmission delay and service queuing time (including microservice execution time). That is, the total delay for business j(n) to complete its reasoning task can be expressed as:

[0087]

[0088] The total latency for business j(n) to complete its reasoning task reflects the efficiency of microservices oriented to business needs.

[0089] Based on the startup time and resource requirements of the microservice deployment described above, as well as the microservice efficiency oriented to business needs, it can be seen that for any microservice i, when starting the deployment, it has the corresponding image layer δ l，i and the lth layer size s l Download requirements; oriented to resource deployment, with computing requirements f i and storage requirements s′ i ; For business service operation, the unit processing computing capacity of microservice i is The output data volume is Further analysis shows that microservice deployment and service methods jointly determine the service efficiency of the business. When the deployment can support centralized processing of multiple microservices involved in the business, the additional delay caused by transmission between nodes can be avoided (local processing has no transmission delay); when resources are scarce or the business volume suddenly increases, low-precision branch microservices can be selected for accelerated processing or collaborative processing by other edge servers or operation centers with sufficient resources.

[0090] Therefore, considering service accuracy and latency, the following overall system optimization goal is given, which is the joint optimization goal of deployment and orchestration. The joint optimization goal of deployment and orchestration can be calculated based on the following formula:

[0091]

[0092] Where F(ψ) represents the joint optimization objective of deployment and orchestration, ψ is the optimization variable, N represents the number of edge server nodes, U represents the number of branches of the deep neural network model, I represents the number of microservices, ω1, ω2, and ω3 are preset weight parameters, ω1 is the service delay weight parameter, ω2 is the service accuracy weight parameter, ω3 is the deployment delay weight parameter, and j(n) represents the service reached by edge server node n; τ j(n) Indicates the download start time of service j(n), The value of indicates whether to select branch u to process service j(n). If branch u is selected, The value is 1, if branch u is not selected, The value of e is 0; u Indicates the computational accuracy (or reasoning accuracy) of branch u; Indicates the download and startup time required for deploying microservice i on edge server node n.

[0093] The optimization variable ψ can be expressed as:

[0094]

[0095] Among them, {b i，n} represents the deployment strategy (deployment variable) of the microservice on the edge server node, represents the branch selection strategy of the deep neural network model (branch selection variable), Represents the microservice orchestration strategy (reselecting edge server nodes for microservices) strategy (orchestration variable), The value of indicates whether microservice i on edge server node n′ processes business j(n), that is, whether microservice i is deployed to edge server node n′. If the value is 1, it means that the edge server node is replaced (from n to n′), and if the value is 0, it means that the edge server node is not replaced.

[0096] The system optimization model is:

[0097]

[0098] st

[0099]

[0100]

[0101] Among them, C1, C2, C3, C4 and C5 are constraints.

[0102] The joint optimization goal of deployment and orchestration in this model P focuses on minimizing the deployment startup time and minimizing the deep network inference latency and its accuracy error (based on branch accuracy). i，n DNN branch selection and edge server node orchestration (i.e. node selection) Constraints C1 and C2 restrict the computation and memory requirements of the deployment. C3 requires that only the corresponding microservice on a specific node be selected from the available microservices. C4 requires that the service select a branching structure. C5 also ensures queue stability. However, the strong coupling between microservice deployment, branching, and orchestration makes finding an optimal solution difficult. Therefore, in this embodiment of the present invention, a reasonable algorithm is proposed based on the following analysis of the impact of different variables on the objective function.

[0103] For the system optimization problem corresponding to the optimization model (13), the present invention optimizes the variables The interactions among the various elements were analyzed.

[0104] 1) Business-oriented microservice deployment i，n (or b i″，n″ )}analyze:

[0105] Business-oriented microservice deployment can affect microservice deployment startup time and the total delay τ of the inference task j(n) .

[0106] In order to reflect the impact of edge server node deployment on data transmission delay, queuing delay, impact on other nodes, and download delay during switching, the gradient value of the deployment and orchestration joint optimization objective function for the deployment variable of the current microservice i″ is calculated, and the following can be obtained:

[0107]

[0108] in:

[0109]

[0110]

[0111] in, represents the gradient value of the deployment variable of the joint optimization objective function of deployment and orchestration with respect to the edge server node n″ where the current microservice i″ is deployed; represents the transmission delay impact value corresponding to the gradient value; Indicates the corresponding queuing delay impact value; Indicates the corresponding impact value on other edge server nodes; Indicates the corresponding download delay impact value.

[0112] Where n, n′, and n″ all represent edge server nodes; n′ represents the preceding edge server node of edge server node n″; i′ represents the preceding microservice of microservice i″ in service j(n); The value of indicates whether the traffic of service j(n) is processed by microservice i″ on edge server node n″ (that is, whether microservice i″ is deployed on edge server node n″). If so, then The value of is 1; otherwise, The value of λ is 0; j(n) represents the arrival rate of service j(n); represents the output data volume of microservice i′; r represents the relationship between the u-th branch of business j and microservice i′ and microservice i″; n′，n″ represents the download rate between edge server nodes n′ and n″; The value of identifies whether n′ and n″ are the same edge server node. If so, then The value of is 1, otherwise, The value of is 0; Indicates the output data volume of microservice i″; Indicates the mapping relationship value between the u-th branch of business j and microservice i″; represents the average stay time of microservice i″ in the processing queue of edge server node n″; The value of indicates whether the traffic volume of service j(n) is processed by microservice i on edge server node n″. If so, then The value of is 1; otherwise, The value of is 0; Indicates the mapping relationship between the u-th branch of business j and microservice i; b i，n′ The value of identifies whether the edge server node n″ deploys microservice i. If so, its value is 1, otherwise, its value is 0; represents the average stay time of microservice i in the processing queue of edge server node n″; b i″，n′ The value of identifies whether the edge server node n″ deploys microservice i″. If yes, its value is 1; if not, its value is 0; represents the download rate between the starting edge server node and edge server node n″; β l，i″ The value of indicates whether microservice i″ includes the l-th layer image data. If so, its value is 1; if not, its value is 0.

[0113] Specifically, based on the above (Right now ), we can see That is b i，n The value of is monotonically increasing relative to the objective function. Combining the objective functions of multiple businesses, multiple nodes, multiple microservices, and multiple branches, we analyze b i″，n′ Affected by external variables (including other deployment b i，n ,and )

[0114] When it is small, that is, when the business from other edge server nodes and the business forwarded to other edge server nodes are small, smaller;

[0115] When it is smaller, that is, when the residence time (queuing delay) of all microservices i″ run by edge server node n″ is smaller, smaller;

[0116] When it is smaller, change the deployment b i″，n′ When the impact on the running time of other microservices on the edge server node n″ is small, smaller;

[0117] When is smaller, that is, when the download time of deploying microservice i″ on edge server node n″ is smaller, Smaller.

[0118] Based on the above analysis, assuming that all initial deployments are local, then and will be large and it will be difficult to meet conditions C1, C2 and C5. i″，n″ =1 changes to b i″，n″ = 0, will force Change, even The change is to reduce the corresponding (i″, n″) And increase the corresponding nodes (i″, n≠n″) When the increase is less than the decrease, the judgment is b i″，n″ = 0, which means deleting the deployment and orchestrating the corresponding service to other better deployment nodes.

[0119] Based on the above analysis, we can give:

[0120]

[0121] in, is the gradient value of the deployment variable of the joint optimization objective function for all edge server nodes that deploy the current microservice i″, is the corresponding number of deployed nodes, It is the relative value of the gradient of deploying microservice i on node n″ and the average deployment on other nodes, and selects the one with the largest gradient among all deployments. The deployment of (i * , n * ), as a deployment strategy that is needed and can be adjusted, where (i * , n * ) represents the selected microservice unit i * At the edge server node n * It should be noted that to adjust the deployment strategy, the potential need is to adjust the arrangement, that is, to deploy in (i * , n * ) to the microservices on other deployment nodes (i * , n≠n * ),therefore, Not only does it represent a larger The corresponding deployment needs to be cancelled and Smaller, with potential for choreographic adjustments.

[0122] 2) Microservice Orchestration analyze

[0123] Microservices orchestration based on deployment and branching Will affect the total delay τ to complete the reasoning task j(n) , that is, choosing the right edge server nodes and branches will speed up the operation of microservices:

[0124] Based on the joint optimization objective function of deployment and orchestration, we can also analyze The value of the external variable (such as deploy b i，n ,and )

[0125] In order to reflect the impact of the edge server node's transmission rate, processing delay, and overall queuing delay, the gradient value of the deployment and orchestration joint optimization objective for the edge server node orchestration variable of the microservice is calculated, and the following is obtained:

[0126] in:

[0127]

[0128] in, represents the gradient of the optimization objective function with respect to the orchestration variable when orchestrating service j(n) on microservice i″ deployed on edge server node n″; Indicates the transmission rate impact value corresponding to the gradient value; Indicates the corresponding processing delay impact value; Indicates the corresponding overall queuing delay impact value; based on (Right now ), we can also analyze Affected by external variables (such as deployment {b i，n}、 and )

[0129] Impact: When microservice i″ and its preceding and following microservices are completed on the same edge server node in the microservice flow of business j(n), Otherwise, when the front and back microservices are completed on the edge server node with higher transmission rate, smaller;

[0130] Impact: When the processing delay of business j(n) orchestrated on microservice i deployed on node n″ is small, smaller;

[0131] Impact: This indicates that when service j(n) is orchestrated on microservice i″ deployed on node n″, the overall queuing delay will increase;

[0132] Based on the above analysis, when a deployment Updated to Will force Then the microservice (i * , n * ) The originally scheduled service j(n) will need to be readjusted to give:

[0133]

[0134] That is, the microservice i on the microservice flow of business j(n)* , orchestrated to minimize transmission and computation costs ( Minimal) deployment (i * , n ** )superior.

[0135] 3) Business-oriented branch selection Business-oriented branch selection The value of will affect the calculation accuracy e u and the total delay τ to complete its reasoning task j(n) Therefore, to determine the set of microservices for business j(n), the following issues need to be considered:

[0136] When using the early branch exit mechanism, the gradient value of the deployment and orchestration joint optimization target for the DNN branch selection variable is calculated, and the branch selection is optimized based on the gradient value. The gradient value is calculated using the following formula

[0137]

[0138] in:

[0139]

[0140]

[0141] in, represents the gradient value of the selected variable of the u-th branch of the deployment and orchestration joint optimization objective for business j(n), represents the transmission time impact value of the u-th branch of business j(n) affected by microservice deployment, orchestration and branching, represents the service queue time impact value of the u-th branch of business j(n) affected by microservice deployment, orchestration and branching, Represents the impact of the u-th branch of business j(n) on the queuing time of other microservices affected by microservice deployment, orchestration, and branches; Represents the accuracy impact value of the u-th branch of service j(n).

[0142] Based on the gradient value From the calculation formula, we can see that branch u determines The accuracy problem is The branch selection strategy and deployment strategies i，n} and orchestration strategies Co-determine, among other things, the orchestration strategy and branch selection strategy is strongly coupled, i.e., the downgrade branch will make Equivalent deletion (deleting microservices on the microservice flow) or even iterative optimization will trigger {b i，n Considering the feasibility and iteration of the algorithm, assume that the current branch of business j(n) is u j(n) , when the branch exits early for 1≤u j(n) <U, and If not updated, only The changes in each branch are measured The pros and cons, Smaller description Smaller.

[0143] Based on the above three aspects of analysis, we can know that microservice deployment is the premise of orchestration, that is, the decision of which microservices to orchestrate and on which edge server node to orchestrate. Strongly dependent on deployment decisions {b i，n},and Strongly dependent on branch selection decisions At the same time, deployment can be oriented towards orchestration, which can achieve better overall results. Based on this, the present invention decouples the above problem into two mutually iterative problems: deployment optimization and microservice orchestration optimization, and adopts algorithms for deployment iterative deletion, orchestration iterative update, and branch demotion respectively.

[0144] As an example, the following algorithm can be used to perform joint optimization of deployment, orchestration, and branch selection.

[0145]

[0146] In this algorithm, the deployment decision {b i，n} and server node orchestration The strong coupling of b i，n Eliminate (i.e., delete the deployment of the microservice on the edge server node) to trigger Adjustment, given b i，n and In the early stages of deployment, iterative elimination may induce Therefore, a trigger flag parameter is introduced here: F lag Parameters, when only b i，n and When the search fails to optimize the deployment and orchestration joint optimization goals, F lag =1 to trigger The branches are downgraded to participate in the iterative algorithm to optimize the goal.

[0147] Based on the above algorithm, it can be seen that the deployment and orchestration joint optimization step S120 of the present invention is specifically as follows: determining the deployment and orchestration joint optimization target based on the microservice deployment startup time, business latency and business accuracy, performing joint optimization including microservice deployment optimization and microservice orchestration optimization according to the deployment and orchestration joint optimization target, and when the result of the joint optimization cannot achieve the deployment and orchestration joint optimization target, further triggering the orchestration iterative adjustment based on the multi-branch deep neural network model branch optimization to achieve the deployment and orchestration joint optimization target.

[0148] Among them, the microservice deployment optimization includes microservice deployment reduction; the microservice orchestration optimization includes replacing the edge server nodes of the microservice deployment.

[0149] In the embodiment of the present invention, performing the joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the deployment and orchestration joint optimization goal may include:

[0150] Calculate the gradient sum of the deployment and orchestration joint optimization objective function on all edge server nodes where the microservice is deployed, and determine the number of all edge server nodes where the microservice is deployed;

[0151] Calculating a relative gradient value of deploying a microservice on an edge server node and an average deployment on other edge server nodes based on the calculated gradient sum and the determined quantity, and determining a microservice deployment reduction strategy by selecting a deployment having the largest relative gradient value among all deployments; and

[0152] The orchestration optimization includes: orchestrating the microservices on the microservice flow of the business to the edge server node deployment with the lowest transmission and computing costs through orchestration update based on the microservice deployment reduction strategy.

[0153] When the result of the joint iterative optimization cannot achieve the deployment and orchestration joint optimization goal, further performing orchestration iterative adjustment based on branch optimization of a multi-branch deep neural network model may include:

[0154] When the target value of the joint iterative optimization no longer decreases and the branch optimization constraints are met, the multi-branch deep neural network model branch iterative optimization is triggered, and the orchestration update is performed after each branch optimization, so as to orchestrate the microservices on the microservice flow of the business to the edge server node deployment with the lowest transmission and computing cost through the orchestration update until the deployment and orchestration joint optimization goal is achieved.

[0155] More specifically, the step of calculating the gradient value of the deployment and orchestration joint optimization objective function for the deployment variable of the current microservice includes: calculating the gradient value of the deployment and orchestration joint optimization objective function for the deployment variable on each edge server node where the current microservice is deployed And based on the calculated gradient value, the total gradient value of the deployment variable of all edge server nodes where the current microservice is deployed is calculated for the deployment and orchestration joint optimization objective function.

[0156] The method of orchestrating the microservices on the microservice flow of the business to the edge server node with the minimum transmission and computing cost through orchestration update includes: calculating the gradient value of the corresponding deployment and orchestration joint optimization target for the microservice orchestration based on the orchestration update decision, selecting the orchestration update decision corresponding to the minimum gradient value to obtain the edge server node with the minimum transmission and computing cost, and orchestrating the microservices on the microservice flow of the business to the selected server node.

[0157] In order to flexibly, quickly and efficiently deploy AI applications at the edge of the network, the present invention uses virtualization technology to allocate and manage resources, uses container virtualization as an operating system-level technology, provides an operating system kernel for each container, and enables an independent container engine. Compared with traditional hardware virtualization, containers have the advantages of low resource consumption and fast startup. Therefore, given that edge computing resources are limited and latency requirements are strict, the use of container virtualization can improve resource utilization and more efficiently meet quality of service (QoS) requirements. The present invention introduces container-based microservices, combines the multi-branch structure of deep neural networks with decomposable microservices, and uses multiple independent microservices to complete complex deep neural network reasoning calculations. By optimizing the deployment of microservices on multiple edge computing platforms and completing the orchestration of microservices for corresponding tasks, the real-time and effectiveness of computing are improved under the reasonable allocation and scheduling of limited resources.

[0158] The solution of the present invention achieves the following functions: 1) deploying microservices on multiple edge servers based on the container image layer, analyzing the storage and time costs of deployment, as well as the operating efficiency under different deployment conditions; 2) supporting DNN applications of AI services by building multi-branch DNNs to meet different service accuracy and latency requirements, and using directed acyclic graphs (DAGs) to define the service process of each branch; 3) based on deployment scenarios, using queuing theory to analyze the computing and residence (queuing) time of different branch models under edge collaboration conditions; 4) the present invention proposes the joint optimization problem of microservice deployment, orchestration and DNN branches, and proposes an effective solution.

[0159] An embodiment of the present invention also provides a microservice deployment and orchestration optimization system for a deep neural network, which includes a computer device, wherein the computer device includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0160] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned method for microservice deployment and orchestration optimization of a deep neural network. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0161] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the aforementioned deep neural network microservice deployment and orchestration optimization method.

[0162] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0163] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0164] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0165] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for microservice deployment and orchestration optimization of deep neural networks, characterized in that: This method is based on a distributed architecture, which includes a cloud center node and an edge server node in communication with the cloud center node. The method includes: Initial deployment step: The cloud center node deploys multiple microservices composed of different container images to multiple edge server nodes based on the microservice deployment startup time and resource conditions. Each microservice is built based on the business branch of the pre-trained multi-branch deep neural network model, and each business branch corresponds to a set of microservices. Deployment and orchestration joint optimization step: determine the deployment and orchestration joint optimization target based on the microservice deployment startup time, business latency and business accuracy, perform joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the deployment and orchestration joint optimization target, and when the result of the joint iterative optimization cannot continue to optimize the deployment and orchestration joint optimization target, further perform orchestration iterative adjustment based on multi-branch deep neural network model branch optimization to achieve the deployment and orchestration joint optimization target; wherein, the microservice deployment optimization includes microservice deployment reduction, and the microservice orchestration optimization includes replacing the edge server nodes of the microservice deployment; the deployment and orchestration joint optimization target is constructed with the goal of reducing the total latency caused by microservice deployment startup and business execution under the premise of meeting business accuracy.

2. The method for microservice deployment and orchestration optimization of a deep neural network according to claim 1, characterized in that: The microservice flow of each business branch is built based on a directed acyclic graph; The multi-branch deep neural network model branch optimization is a branch optimization algorithm based on a branch early exit algorithm.

3. The method for microservice deployment and orchestration optimization of a deep neural network according to claim 1, characterized in that: The method of deploying multiple microservices composed of different container images to multiple edge server nodes based on the microservice deployment startup time and resource conditions includes: Pull the container image corresponding to each microservice from the image layer set in the image repository, download the pulled container image to the matching edge server according to the pre-set constraints, and deploy multiple microservices composed of different container images to multiple edge server nodes; The constraints include: the startup time of deploying the microservice is less than a predetermined image download delay threshold; the remaining computing resources and storage resources of the edge server node where the microservice is to be deployed are greater than the computing resources and storage resources required by the microservice.

4. The method for microservice deployment and orchestration optimization of a deep neural network according to claim 1, characterized in that: The deployment and orchestration joint optimization goal is determined based on the following formula: ； in, represents the joint optimization goal of deployment and orchestration, To optimize the variables, Indicates the number of edge server nodes, Indicates the number of branches of the deep neural network model, Indicates the number of microservices, is the service delay weight parameter, is the precision weight parameter of the business, is the weight parameter of deployment delay, Represents an edge server node The business reached; Indicates business Download start time, The value of indicates whether to select a branch of the deep neural network model Handling business , Represents branches of a deep neural network model The calculation accuracy of Representing microservices At the edge server node The download startup time required for deployment.

5. The method for microservice deployment and orchestration optimization of a deep neural network according to claim 4, characterized in that: The optimization variables are: ； in, Represents the deployment decision of the microservice on the edge server node, represents the branch selection decision of the deep neural network model, Represents microservice orchestration decisions, The value indicates whether the edge server node Microservices on Handling business .

6. The method for microservice deployment and orchestration optimization of a deep neural network according to any one of claims 1 to 5, characterized in that: The joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the deployment and orchestration joint optimization goal includes: Calculate the gradient of the deployment and orchestration joint optimization objective function for the deployment variables of the current microservice; Based on the calculated gradient value and the number of edge server nodes where the current microservice is deployed, a relative gradient value between deploying the current microservice on one edge server node and the average deployment on other edge server nodes is calculated, and a microservice deployment reduction strategy is determined by selecting the deployment with the largest relative gradient value among all deployments. The joint iterative optimization includes: orchestrating the microservices on the microservice flow of the business to the edge server node deployment with the minimum transmission and computing cost through orchestration update based on the microservice deployment reduction strategy; When the result of the joint iterative optimization cannot achieve the deployment and orchestration joint optimization goal, further performing orchestration iterative adjustment based on branch optimization of the multi-branch deep neural network model, including: When the optimization target value of the joint iterative optimization no longer decreases and the branch optimization constraint conditions are met, the multi-branch deep neural network model branch iterative optimization is triggered, and the orchestration update is performed after each branch optimization, so as to orchestrate the microservices on the microservice flow of the business to the edge server node deployment with the lowest transmission and computing cost through the orchestration update until the deployment and orchestration joint optimization goal is achieved.

7. The method for microservice deployment and orchestration optimization of a deep neural network according to claim 6, characterized in that: The step of calculating the gradient value of the deployment and orchestration joint optimization objective function for the deployment variable of the current microservice includes: calculating the gradient value of the deployment and orchestration joint optimization objective function for the deployment variable of each edge server node where the current microservice is deployed, and calculating the total gradient value of the deployment and orchestration joint optimization objective function for the deployment variable of all edge server nodes where the current microservice is deployed based on the calculated gradient value; The method of orchestrating the microservices on the microservice flow of the business to the edge server node deployment with the minimum transmission and computing cost through orchestration update includes: calculating the gradient value of the corresponding deployment and orchestration joint optimization target for the microservice orchestration variable based on the orchestration update decision, selecting the orchestration update decision corresponding to the minimum gradient value to obtain the edge server node with the minimum transmission and computing cost, and orchestrating the microservices on the microservice flow of the business to the selected server node.

8. The method for microservice deployment and orchestration optimization of a deep neural network according to claim 1, characterized in that: In the initial deployment step, the front and back microservices on the microservice flow are deployed on the nearest edge server node, and the longest branch in the multi-branch deep neural network model is used as the currently selected branch.

9. A deep neural network microservice deployment and orchestration optimization system, characterized in that: The system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method according to any one of claims 1 to 8.

Images