Micro-service deployment and arrangement optimization method and system of deep neural network

Abstract

The invention provides a micro-service deployment and arrangement optimization method and system for a deep neural network, and the method comprises the steps: enabling a cloud center node to deploy a plurality of micro-services composed of different container mirror images to a plurality of edge server nodes based on the micro-service deployment starting time and resource conditions, each micro-service is constructed based on a service branch of a pre-trained multi-branch deep neural network model, and each service branch corresponds to a micro-service set; determining a deployment and arrangement joint optimization target based on micro-service deployment starting time, service time delay and service precision, and performing joint iterative optimization including micro-service deployment optimization and micro-service arrangement optimization according to the deployment and arrangement joint optimization target. And when the result of the joint iterative optimization cannot achieve the deployment and arrangement joint optimization target, further performing arrangement iterative adjustment based on multi-branch deep neural network model branch optimization to achieve the deployment and arrangement joint optimization target.

Description

Technical Field

[0001] The present invention relates to the technical field of the Internet of Things, and particularly to a method and system for optimizing the microservice deployment and orchestration of a deep neural network. Background Art

[0002] The rapid development of the Internet of Things (IoT) has brought about the need for generating and processing massive amounts of data. In-depth research on big data and artificial intelligence (AI), as well as the implementation of efficient computing technologies, have become the key to dealing with these data.

[0003] The progress of deep learning has significantly promoted the development and application of AI, achieving human-level accuracy in fields such as computer vision and speech recognition. The neuron weights in a deep neural network (DNN) are optimized through data training to meet specific application requirements and tasks. However, this high accuracy brings huge computational and memory requirements. To address resource limitations, researchers utilize the deep structure characteristics of DNNs, combining accuracy requirements and computational consumption to compress the model. Research shows that neural networks with different structures can be used for inference, resulting in different computational loads and accuracies. Using shallow neural networks can capture most of the information, effectively reducing the computational load and latency while maintaining the accuracy requirements.

[0004] Edge computing brings computing and data storage closer to the location of demand, avoiding the network bottleneck at the central location of cloud computing. This is particularly important for emerging real-time AI applications, especially those driven by deep neural networks in intelligent environments that require immediate decision-making. In addition, distributing the computational load to multiple edge server nodes improves the network scalability through horizontal scaling, resource optimization, and geographical distribution. These advantages make edge computing an important part of modern IT infrastructure, capable of supporting a wide range of applications with high performance and high reliability.

[0005] However, since the training and inference of deep learning consume a large amount of computing resources, and the computational effectiveness directly affects the real-time performance and effectiveness of artificial intelligence applications, how to combine the current edge computing with real-time performance and scalability to improve the real-time performance and effectiveness of deep neural network inference applications under limited edge resource conditions is an urgent problem to be solved. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a method for optimizing the microservice deployment and orchestration of a deep neural network to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of the present invention provides a method for optimizing the microservice deployment and orchestration of a deep neural network. This method is based on a distributed architecture, which includes a cloud center node and edge server nodes communicatively connected to the cloud center node. The method includes: Initial deployment step: The cloud center node deploys multiple microservices composed of different container images to multiple edge server nodes based on the microservice deployment start time and resource conditions. Among them, each microservice is constructed based on a business branch of a pre-trained multi-branch deep neural network model, and each business branch corresponds to a microservice set; Joint optimization step of deployment and orchestration: Determine the joint optimization goal of deployment and orchestration based on the microservice deployment start time, service latency, and service accuracy. Perform joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the joint optimization goal of deployment and orchestration. When the result of the joint iterative optimization cannot achieve the joint optimization goal of deployment and orchestration, further perform an orchestration iterative adjustment based on the branch optimization of the multi-branch deep neural network model to achieve the joint optimization goal of deployment and orchestration.

[0008] In some embodiments of the present invention, the microservice flow of each business branch is constructed based on a directed acyclic graph; the branch optimization of the multi-branch deep neural network model is a branch optimization algorithm based on the branch early exit algorithm.

[0009] In some embodiments of the present invention, the step of deploying multiple microservices composed of different container images to multiple edge server nodes based on the microservice deployment start time and resource conditions includes: pulling the container images corresponding to each microservice from the image layer set in the image repository, and downloading the pulled container images to the matching edge servers according to the pre-set constraint conditions, so as to deploy multiple microservices composed of different container images to multiple edge server nodes; the constraint conditions include: the start time of deploying the microservice is less than a predetermined image download latency threshold; the remaining computing resources and storage resources of the edge server node where the microservice is to be deployed are greater than the computing resources and storage resources required by the microservice.

[0010] In some embodiments of the present invention, the microservice deployment optimization includes microservice deployment reduction; the microservice orchestration optimization includes replacing the edge server nodes where the microservices are deployed.

[0011] In some embodiments of the present invention, the joint optimization goal of deployment and orchestration is determined based on the following formula: where F(ψ) represents the joint optimization goal of deployment and orchestration, ψ is the optimization variable, N represents the number of edge server nodes, U represents the number of branches of the deep neural network model, I represents the number of microservices, ω1 is the latency weight parameter of the service, ω2 is the accuracy weight parameter of the service, ω3 is the weight parameter of the deployment latency, j(n) represents the service reached by the edge server node n; τ j(n) represents the download start time of the service j(n), The value indicates whether to select branch u of the deep neural network model to process service j(n), e u represents the computing precision of branch u of the deep neural network model; represents the download startup time required for the deployment of microservice i on edge server node n.

[0012] The optimization variables are: where {b i，n} represents the deployment decision (deployment variable) of the microservice on the edge server node, represents the branch selection decision (branch selection variable) of the deep neural network model, represents the microservice orchestration decision (orchestration variable), The value indicates whether to process service j(n) of microservice i on edge server node n'.

[0013] In some embodiments of the present invention, the joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the joint optimization objective of deployment and orchestration includes: Calculating the gradient value of the joint optimization objective function of deployment and orchestration with respect to the deployment variable of the current microservice; Based on the calculated gradient value and the number of edge server nodes for deploying the current microservice, calculating the relative gradient value of deploying the current microservice on one edge server node and the average deployment on other edge server nodes, and determining the microservice deployment reduction strategy by selecting the deployment with the largest relative gradient value in all deployments; The joint iterative optimization includes: based on the microservice deployment reduction strategy, through orchestration update, orchestrating the microservices on the microservice flow of the service to the edge server node deployment with the minimum transmission and computing cost; When the result of the joint iterative optimization cannot achieve the joint optimization objective of deployment and orchestration, further performing an orchestration iterative adjustment based on the branch optimization of the multi-branch deep neural network model, including: when the optimization objective value of the joint iterative optimization no longer decreases and satisfies the branch optimization constraint conditions, triggering the branch iterative optimization of the multi-branch deep neural network model, and performing orchestration update after each branch optimization, so as to orchestrate the microservices on the microservice flow of the service to the edge server node deployment with the minimum transmission and computing cost until the joint optimization objective of deployment and orchestration is achieved.

[0014] In some embodiments of the present invention, the steps of calculating the gradient value of the computing deployment and orchestration joint optimization objective function for the deployment variables of the current microservice include: calculating the gradient value of the computing deployment and orchestration joint optimization objective function for the deployment variables of each edge server node where the current microservice is deployed, and calculating the total gradient value of the computing deployment and orchestration joint optimization objective function for the deployment variables of all edge server nodes where the current microservice is deployed based on the calculated gradient values; The method of orchestrating the microservices on the microservice flow of the service to the edge server node deployment with the minimum transmission and computing cost through orchestration updates includes: for the orchestration update decision, calculating the gradient value of the corresponding computing deployment and orchestration joint optimization objective for the microservice orchestration variables, selecting the orchestration update decision corresponding to the minimum gradient value to obtain the edge server node with the minimum transmission and computing cost, and orchestrating the microservices on the microservice flow of the service to the selected server node.

[0015] In some embodiments of the present invention, in the initial deployment step, the front and rear microservices on the microservice flow are deployed on the nearest edge server nodes, and the longest branch in the multi-branch deep neural network model is used as the currently selected branch.

[0016] Adopting the above solution, container-based microservices are introduced, combining the branch structure of the deep neural network with decomposable microservices, using multiple independent microservices to complete complex deep neural network inference calculations, and by optimizing the microservice deployment on multiple edge computations and the microservice orchestration for completing corresponding tasks, the real-time performance and effectiveness of the computation are improved under the reasonable allocation and scheduling of limited resources.

[0017] The second aspect of the present invention further provides a microservice deployment and orchestration optimization system for a deep neural network. The system includes a computer device, the computer device includes a processor and a memory, computer instructions are stored in the memory, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0018] The third aspect of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps implemented by the foregoing microservice deployment and orchestration optimization method for a deep neural network.

[0019] The additional advantages, objectives, and features of the present invention will be partially elaborated in the following description, and will become partially apparent to those of ordinary skill in the art after studying the following text, or can be learned through the practice of the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the structures specifically pointed out in the specification and the drawings.

[0020] Those skilled in the art will understand that the purposes and advantages achievable by the present invention are not limited to those specifically described above, and the above and other purposes achievable by the present invention will be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not limit the present invention.

[0022] Figure 1 It is a schematic diagram of the method for optimizing the microservice deployment and orchestration of a deep neural network in an embodiment of the present invention.

[0023] Figure 2 It is a schematic diagram of collaborative microservice deployment and orchestration in an embodiment of the present invention.

[0024] Figure 3 It is a schematic diagram of the relationship between the CPU occupancy rate and the computing time of an edge server node in an embodiment of the invention.

[0025] Figure 4 It is a schematic diagram of a multi-branch DNN structure in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In order to make the purposes, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the embodiments and the drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not limit the present invention.

[0027] Here, it should also be noted that in order to avoid obscuring the present invention due to unnecessary details, only the structures and / or processing steps closely related to the solution according to the present invention are shown in the drawings, and other details less related to the present invention are omitted.

[0028] To flexibly, quickly, and efficiently deploy AI applications at the network edge and achieve the objectives of the present invention, the present invention adopts container virtualization technology and proposes an optimization method for the deployment and orchestration of microservices (MS) of deep neural networks. The container virtualization technology enables an independent container engine to encapsulate an application program and its dependencies in a container, allowing the application program and its dependencies to run in an isolated environment. In the deployment of an architecture based on container virtualization, microservices, as a key technology, have the advantage of loose coupling. By deploying container-based microservices on multiple edge server nodes, the deployment architecture of the present invention can decompose complex applications into smaller, interconnected components, and each component runs its own microservice. By encapsulating each service in its own container, microservices can be developed, deployed, and scaled independently. In addition, when there are overlapping microservices in an application, shared components can be used, and each component can be updated independently without affecting other components. Therefore, the container-based microservice deployment can not only meet the latency requirements for resource and service deployment in edge computing but also enhance the elasticity and scalability of services.

[0029] The method of the present invention is based on a distributed architecture. As Figure 2 shown, the distributed architecture may include 1 cloud center server node (hereinafter referred to as the cloud center node) and several (such as N, N = {n, n = 1,... n,... N}) edge server nodes (or simply referred to as edge nodes, EN) communicatively connected to the cloud center node. The resource capacity of the cloud center node is stronger than that of the edge server nodes. The cloud center node deploys I container-based microservices (microservice set I = {i, i = 1,... I}) on each edge server node based on deep neural networks, edge computing, and container-based microservice technology. The i-th microservice is denoted as MSi in Figure 2 . As Figure 2 shown, the microservice MSi is initially deployed on the n-th edge server node (ENn) based on the microservice deployment startup time (image pulling latency, or download latency) and the resource conditions of the edge server node. Due to the different amounts of business arrivals at each edge server, it may lead to a too long queuing time for tasks in the service queue of some edge server nodes. At this time, the joint optimization of deployment and orchestration can be performed based on the task queuing latency of the microservice on the edge server node and the data transmission latency between edge server nodes of the microservice flow. In addition, in the process of joint optimization of deployment and orchestration, the early exit mechanism of the branches of the deep neural network can be further combined to improve the optimization, that is, the DNN branches are selected for deployment and orchestration, and an optimal choice is made between the system resource executability and the accuracy requirements. Thus, through the cooperation between edge servers, task requests from users can be processed quickly and efficiently, and the corresponding deep neural network inference can be completed. Figure 2In this context, i, i', and i'' have the same meaning, all representing microservices. The different symbols are only for distinction, indicating different microservices. Similarly, n, n', and n'' also have the same meaning, all representing edge server nodes. The different symbols are only for distinction, indicating possibly different edge server nodes.

[0030] As Figure 1 shown, the microservice deployment and orchestration optimization method for the deep neural network of the present invention includes a microservice initial deployment step S110 and a deployment and orchestration joint optimization step S120.

[0031] In the initial deployment step S110, the cloud center node deploys multiple microservices composed of different container images to multiple edge server nodes based on the microservice deployment start time and resource conditions.

[0032] The cloud center node constructs a pre-trained multi-branch DNN model, and all container images are deployed in the image repository of the cloud center node. All the images in the repository can be equivalently regarded as a multi-layer (such as L-layer) image structure, that is, composed of multiple image layers. The set of image layers can be represented as L = {1,..., L}. To deploy container-based microservices on edge server nodes, it is first necessary to pull the container images from the image repository of the cloud center node and download them to the edge servers, so as to deploy multiple microservices composed of different container images to multiple edge server nodes. Thus, one or more containers can be started using the downloaded images. In the embodiments of the present invention, to solve the problems of real-time performance and effectiveness of deep neural network inference applications, each microservice is constructed based on the business branches of a pre-trained multi-branch deep neural network (DNN) model, and each business branch corresponds to a microservice set.

[0033] During the microservice deployment process, it is necessary to consider the microservice deployment start time and resource conditions of each edge server node. Among them, the microservice deployment start time may include the microservice download start time (also known as the image download delay or pull delay), and the resource conditions may include computing resources and storage resources (such as memory resources and / or cache resources, etc.). That is, in addition to considering the microservice deployment start time, it is also necessary to consider whether the edge server node can support the operation of the microservice.

[0034] Assume that all images are pre-deployed on the cloud center node, and a certain edge server node pre-caches some image layers. If a microservice is to be deployed to this edge server node, β l，i can be used to represent whether the edge server node needs to download the image layer for the microservice. For example, β l，i = 1 means that the image layer l required by the microservice i is not available locally on this edge server node, so it needs to be downloaded; β l，i= 0 indicates that the image layer l required by microservice i is not needed or has been cached on this edge server node, so there is no need to download it. If b i，n = 1 indicates that edge server node n needs to deploy microservice i, then b i，n = 0 can indicate that edge server node n does not need to deploy microservice i. In this case, when deploying microservices composed of multiple image layers to different server nodes, the required image download delay can be expressed as: where b i，n is used to identify whether microservice i is deployed on server n; β l，i is used to identify whether it is necessary to download image layer 1 for microservice i, s1 represents the file size of image layer l, that is, the memory boundary of the l-th layer image data of the microservice, and r n represents the download rate. Among them, the download rate r n can be calculated based on the bandwidth and signal-to-noise ratio from the cloud center node to edge server node n.

[0035] Based on the download start time of the global deployment can also be obtained. The download start time of the global deployment can be expressed as the sum of the download start times required for deploying all microservices on all edge server nodes. For example, the download start time of the global deployment can be expressed as where represents the download start time required for deploying microservice i on edge server node n.

[0036] It can be seen from this that when deploying microservice i with different image compositions to different nodes n, there are different image download delays When performing microservice deployment, in order to ensure the startup efficiency of microservices, it can be set that the microservice startup time (image download delay) deployed on the edge server node should be less than a predetermined time threshold. In the initial deployment step of the present invention, preferably, the microservices are arranged on edge server nodes with relatively small transmission and calculation costs. For example, the front and rear microservices on the microservice flow are deployed on the nearest edge server nodes.

[0037] In addition, assuming that the resource configuration of microservice i requires a CPU occupancy of f i , and the storage resource requirement is s' i , then the total load of the microservices deployed on the edge server node in terms of computing resources and storage resources cannot exceed the corresponding total computing resources and storage resources (such as memory resources) of the node. That is, it is necessary to satisfy: where s' i = ∑ lβ l，i s l ,F n and S n respectively represent the available computing resource boundary and memory resource boundary on the server node n.

[0038] Furthermore, when deploying the microservice i composed of different images to different nodes n, the remaining computing resources and storage resources of the edge server node where the microservice is to be deployed should be greater than the computing resources and storage resources required by the microservice.

[0039] Therefore, in the embodiments of the present invention, based on the microservice deployment startup time and resource conditions, multiple microservices composed of different container images are deployed to multiple edge server nodes, which may specifically include: pulling the container images corresponding to each microservice from the image layer set in the image repository, and downloading the pulled container images to the matching edge servers according to the preset constraint conditions, so as to deploy multiple microservices composed of different container images to multiple edge server nodes; where the constraint conditions may include: the startup time of deploying the microservice is less than a predetermined time threshold, and the remaining computing resources and storage resources of the edge server node where the microservice is to be deployed are greater than the computing resources and storage resources required by the microservice. This constraint condition is only an example, and the present invention is not limited thereto.

[0040] Through the fine-grained structure of microservices, different types of services can dynamically adjust the infrastructure according to needs. For example, the clusters of compute-intensive services and I / O-intensive service clusters are interleaved, which can not only improve the utilization rate of computing resources but also optimize the use of network bandwidth.

[0041] In addition, it should also be noted that, under the condition of not exceeding the total amount of resources, the resource load still has a great impact on the system performance, especially the service execution performance. Here, a parallel I / O benchmark test in a multi-threaded environment is given, and the verification results show that as the CPU occupancy rate increases, the computing time gradually increases, as Figure 3 shown. Taking 5 digital calculation as an example, when the CPU occupancy rate is 80%, the computing time is 6 times that when the occupancy rate is 20%. It can be seen that the resource load rate affects the processing speed of the edge server node.

[0042] Therefore, in the preferred embodiments of the present invention, when deploying microservices, the computing resource occupancy rate (resource load rate) can be further considered to set or optimize the constraint conditions to improve the inference efficiency as much as possible.

[0043] Deep neural network-based application services are significantly different from other services in terms of logical behavior and system resource occupancy. It is difficult for a single edge server node with limited resources to meet the computing timeliness requirements of deep neural network training involved in the business. In the case of deploying microservices of deep neural networks to multiple edge server nodes as described above, the real-time performance and effectiveness of deep neural network inference applications can be greatly improved. However, due to the real-time nature of the applications, there may still be problems such as insufficient computing resources or poor inference efficiency on edge server nodes. For a multi-branch DNN network structure, based on the early exit mechanism of branches, the computational complexity and actuarial accuracy are different when using different network structures for inference. If a partial shallow network is used, most information can still be captured, and the computational complexity and computational latency can be effectively reduced at the cost of minimal accuracy loss. Therefore, in the embodiments of this application, the constructed DNN network structure is a pre-trained multi-branch DNN network structure, as Figure 4 shown (for example only), so that the computational requirements can be dynamically adjusted based on the computational complexity of the input data and the business accuracy requirements, and the computing can be saved by selectively activating its internal modules (selecting appropriate branches), improving the inference efficiency, and thus different service accuracy and latency requirements can be met. However, how to further optimize considering microservice deployment, orchestration optimization, and DNN branch structure is a difficult problem to solve due to the strong coupling relationship among microservice deployment, DNN branch selection, and orchestration, which is also the problem that the present invention will further solve.

[0044] In the microservice deployment stage of the present invention, each microservice is constructed based on the business branches of a pre-trained multi-branch deep neural network model, and each business branch corresponds to a microservice set. More specifically, based on the container architecture, the microservice flows of different DNN business branches can be constructed based on a directed acyclic graph (DAG).

[0045] More specifically, the u-th branch of a specific DNN business is represented by graph G u =(I u , ε u ), where u = 1,..., U, and U is the number of branches of this business. The microservice set of the u-th branch is I u , as the nodes of the DAG, and the matrix is the microservice association relationship on the u-th branch of this business, that is, the set of edges in the DAG. Its expanded form is: where I u =|I u |, and ε u，v，v′ ∈{0, 1} is the element in the v-th row and v'-th column of ε u , which is used to identify that the u-th branch of the business is based on the microservice set Iu Microservice association relationship; v, v′=1,…,I u ;i u，v Indicates I u The vth element of . When δ u，v，v′ =1 indicates that there is a dependency relationship between microservices, that is, microservice i u，v ∈I u It is a microservice u，v′ ∈I u Pre-order operation, microservice i u，v The output of microservice i u，v′ Input; when ε u，v，v′ = 0, microservice i u，v and microservices u，v′ There is no dependency between them.

[0046] The pre-trained multi-branch DNN model has a branch early exit mechanism for different businesses. It obtains business accuracy and corresponding computing requirements, and selects different inference accuracy e according to different computing requirements, latency requirements, etc. u (mean square loss) branch u, where u≤U and satisfies |I u |<|I u+1 |≤I,e u >e u+1 Preferably, in the initial deployment step, when performing branch selection, the longest branch among the DNN branches is preferentially selected as the current branch by default. In the subsequent optimization process, branch selection can be performed based on the early branch withdrawal mechanism.

[0047] In order to facilitate the microservice collection of business branches I u The mapping between the complete set of microservices I is given on the basis of I = {i, i = 1, ..., I} u and I u =ξ u I (4) Among them, ξ u For I u and the mapping relationship matrix of I, ξ u，v，i Yes u The element in row v and column i of u，v =i, ξ u，v，i =1, otherwise ξ u，v，i = 0. Let ξ u，i for ξ u The i-th column of u，i =0, indicating that microservice i does not exist in I u At this time, you can identify the quantity Indicates the mapping relationship value between branch u and microservice i, Indicates that microservice i does not exist in branch u, otherwise

[0048] Based on I u and the mapping relationship between I and G u =(I u , ε u ), the extended graph parameters can be obtained where is the element in the i-th row and i'-th column of, which is used to represent the microservice association relationship of the u-th branch of the service based on the microservice set I; if and only if ε u，v，v′ = 1, and i = i u，v , i' = i u，v′ at this time, indicates that the microservices i and i' in the u-th branch of the service are sequentially associated; otherwise

[0049] After the microservices corresponding to the service branches of the pre-trained multi-branch deep neural network model are deployed to multiple edge server nodes through the above microservice deployment process, the deep neural network model can be efficiently applied for inference calculation.

[0050] However, since artificial intelligence applications are real-time, it is easy to have situations such as resource shortage or calculation efficiency decline due to the sudden increase in business volume on each edge server node. At this time, for the inference tasks of multi-branch DNN, a suitable microservice orchestration mechanism is required to cooperate to meet the business accuracy and latency requirements. That is, it is necessary to perform microservice orchestration optimization during deployment to further improve the real-time performance and effectiveness of the deep neural network inference application.

[0051] Therefore, the microservice deployment and orchestration optimization method of the deep neural network of the present invention further includes a joint optimization step S120 of deployment and orchestration based on the initial deployment result of step S110. The implementation of the joint optimization of deployment and orchestration of the present invention will be described below.

[0052] The factors affecting the real-time performance and effectiveness of the deep neural network inference application may include the microservice deployment start time, the latency of the service, and the business accuracy. Among them, the microservice deployment start time includes the image pulling and downloading latency, and the latency of the service may include the latency such as microservice execution latency, transmission latency, and service queuing latency, which affects the microservice efficiency facing business requirements; the business accuracy is related to the branch selection of the multi-branch DNN structure, and different business accuracies correspond to different calculation loads. The multi-branch DNN will obtain different inference accuracies and calculation requirements due to different early exit mechanisms.

[0053] In an embodiment of the present invention, the joint optimization objective of deployment and orchestration can be determined based on the startup time of microservice deployment, the latency of the service, and the service accuracy, and the joint optimization of microservice deployment and orchestration of the entire system can be performed based on the determined joint optimization objective of deployment and orchestration. The purpose of this joint optimization of deployment and orchestration is to minimize the total latency caused by microservice deployment startup and service execution as much as possible while meeting the service accuracy, thereby improving the microservice efficiency for business requirements.

[0054] Next, the factors affecting the microservice efficiency for business requirements are analyzed.

[0055] The arrival time and arrival volume of services within the coverage range of each edge server node are different. Considering the principle of the nearest access to the network, in the present invention, the traffic volume of service j(n) arriving at edge server node n is modeled as following a Poisson distribution with arrival rate λ j (n), and the service size (computing requirement) follows an exponential distribution with mean μ -1 , that is, the service is modeled as conforming to the M / M / 1 model. Here, it is considered that services are generated at the edge side, and the cloud center node does not have initial arriving services, but can cooperate to process other edge services.

[0056] Taking the single-instance model as an example, for service j(n) arriving at edge server node n, if is used to represent whether to select the u-th branch of the multi-branch DNN to process service j(n), then when the u-th branch is selected, the microservice flow can be sequentially utilized to complete service j(n) according to the expansion graph parameters of the microservice flow . If is used to represent whether service j(n) is processed by microservice i on server node n′, when it is completed by microservice i orchestrated on server node n′ (that is, the corresponding microservice is deleted from the original edge server node and orchestrated to other better edge server nodes), otherwise

[0057] Regarding the microservice flow structure of service j(n), considering the handover process between the previous and the next microservices, that is, the handover from the previous microservice i′ (deployed on edge server node n′) to microservice i″ (deployed on edge server node n″), i″≠i′ and when microservice i″ and microservice i″ are both deployed on node n′ (n′ = n″) and operate efficiently, then there is and that is, there is no data transmission time required to run microservice i″, that is, the transmission latency when running microservice i″ Otherwise, when microservice i″ is not deployed on edge server node n′ or it is deployed but has a heavy running load, microservice i′ needs to be transferred to node n″ with microservice i″ to provide services after it finishes on node n′ (n″≠n′). At this time, and Taking microservice i″ as the general observation object, the average transmission time of the data it needs to process is: where r n′，n″ represents the transmission rate from edge server node n′ to edge server node n″. The value of is used to indicate whether server nodes n′ and n″ are the same node. When and only when n′≠n″, otherwise that is, there is no transmission delay when running on the same node. represents the unit processing computation of microservice i′, represents the output data volume of microservice i′. The parameters and are defined in the same way as , except that different subscripts are used to indicate microservice switching and possible node switching; λ j(n) represents the traffic volume of service j(n); represents whether microservices i′ and i″ in the u-th branch of this service are associated in sequence. b i′，n′ and b i″，n′ are defined in the same way as b i，n , representing whether server nodes n, n′ and n″ are deployed with microservices i, i′ and i″ respectively.

[0058] Considering the microservice queue on node n″ and adopting the first-in-first-out (FIFO) principle, the average sojourn time of microservice i″ in the queue on node n″ can be expressed as: where, is the average value of the queue occupancy ratio of microservice i″ on node n″, is the average service time of microservice i″ on node n″. According to the modeling characteristics of service j(n) of DNN, the service model of each microservice is an M / M / 1 model. Considering the selection and offloading of microservices between nodes, and according to its queue characteristics and statistical characteristics, the average sojourn time of microservice i″ in the queue on node n″ can be expressed as: where: Among them, is the business arrival rate of microservice i″ on edge server node n″; ζ n″ is the processing speed of edge server node n″; is the business computing requirement parameter.

[0059] Then the queuing time for business j(n) to perform microservice i″ is: The total delay for business j(n) to complete its inference task needs to comprehensively consider the microservice transmission delay and service queuing time (including microservice execution time), that is, the total delay for business j(n) to complete its inference task can be expressed as: The total delay for business j(n) to complete its inference task reflects the microservice efficiency for business requirements.

[0060] Based on the startup time and resource requirements of the microservice deployment described above, and the microservice efficiency for business requirements, it can be known that for any microservice i, when starting the deployment, it has a download requirement for the corresponding image layer δ l，i and the size s of the l-th layer l ; for resource deployment, it has a computing requirement f i and a storage requirement s′ i ; for business service operation, the unit processing computing amount of microservice i is The output data volume is Further analysis shows that the microservice deployment and service mode jointly determine the service efficiency of the business. When the deployment can support the centralized processing of multiple microservices involved in the business, the additional delay caused by transmission between nodes can be avoided (local processing has no transmission delay); when resources are scarce or the business volume suddenly surges instantaneously, microservices with low-precision branches can be selected for accelerated processing or collaborative processing by other edge servers or operation centers with sufficient resources.

[0061] Therefore, considering business accuracy and delay, the following overall system optimization objective is given, that is, the joint optimization objective of deployment and orchestration, and the joint optimization objective of deployment and orchestration can be calculated based on the following formula: Among them, F(ψ) represents the joint optimization objective of deployment and orchestration, ψ is the optimization variable, N represents the number of edge server nodes, U represents the number of branches of the deep neural network model, I represents the number of microservices, ω1, ω2, and ω3 are preset weight parameters, ω1 is the delay weight parameter of the business, ω2 is the accuracy weight parameter of the business, ω3 is the weight parameter of the deployment delay, and j(n) represents the business arriving at edge server node n; τ j(n) represents the download startup time of business j(n), The value indicates whether to select branch u to process service j(n). If branch u is selected, then the value is 1. If branch u is not selected, then the value is 0; e u represents the computing accuracy (or called inference accuracy) of branch u; represents the download startup time required for microservice i to be deployed on edge server node n.

[0062] The optimization variable ψ can be expressed as: where, {b i，n} represents the deployment strategy (deployment variable) of the microservice on the edge server node, represents the branch selection strategy (branch selection variable) of the deep neural network model, represents the microservice orchestration strategy (reselecting the edge server node for the microservice) strategy (orchestration variable), The value of indicates whether to process service j(n) of microservice i on edge server node n′, that is, whether to deploy microservice i to edge server node n′. If the value is 1, it means that the edge server node is replaced (replaced by n with n′). If the value is 0, it means that the edge server node is not replaced.

[0063] The system optimization model is: s.t. where, C1, C2, C3, C4, and C5 are constraint conditions.

[0064] In the deployment and orchestration joint optimization objective in this model P, it not only focuses on minimizing the deployment startup time, but also minimizes the deep network inference latency and its accuracy error (based on branch accuracy). In the optimization process, it involves microservice deployment {b i，n}, DNN branch selection and edge server node orchestration (i.e., node selection) where, constraint conditions C1 - C2 constrain the computing and memory of the deployment. C3 requires selecting and only selecting the corresponding microservice on a certain node among the optional microservices. C4 requires the service to select a branch structure. C5 meets the queue stability requirement. However, due to the strong coupling relationship between the microservice deployment, branch, and orchestration problems, it makes the optimal solution difficult. Therefore, in the embodiments of the present invention, based on the following analysis of the influence of different variables on the objective function, a reasonable algorithm is given.

[0065] For the system optimization problem corresponding to the optimization model (13), the present invention analyzes the interactions among the elements of the optimization variables .

[0066] 1) Analysis of business-oriented microservice deployment {b i，n (or b i″，n″ )}: Business-oriented microservice deployment can affect the startup time of microservice deployment and the total latency τ of the inference task j(n) .

[0067] To reflect the influence values of the deployment of edge server nodes on data transmission latency, queuing latency, influence on other nodes, and download latency during handover, calculate the gradient value of the deployment and orchestration joint optimization objective function for the deployment variables of the current microservice i″, then we can obtain: Where: Wherein, represents the gradient value of the deployment and orchestration joint optimization objective function for the deployment variables of the edge server node n″ where the current microservice i″ is deployed; represents the corresponding influence value of transmission latency in the gradient value; represents the corresponding influence value of queuing latency; represents the corresponding influence value on other edge server nodes; represents the corresponding influence value of download latency.

[0068] Wherein, n, n′, and n″ all represent edge server nodes; n′ represents the previous edge server node of the edge server node n″; i′ represents the previous microservice of the microservice i″ in the service j(n); The value of represents whether the traffic of service j(n) is processed on the microservice i″ on the edge server node n″ (i.e., whether to deploy the microservice i″ on the edge server node n″). If so, the value of is 1; if not, the value of is 0; λ j(n) represents the arrival rate of service j(n); represents the output data volume of the microservice i′; represents the relationship value between the u-th branch of service j and the microservices i′ and i″; r n′，n″ represents the download rate between the edge server nodes n′ and n″; The value identifies whether n' and n" are the same edge server node. If so, the value is 1. If not, the value is 0; represents the output data volume of microservice i"; represents the mapping relationship value between the u-th branch of service j and microservice i"; represents the average sojourn time of microservice i" in the processing queue of edge server node n"; The value indicates whether the traffic volume of service j(n) is processed by microservice i on edge server node n". If so, the value is 1; if not, the value is 0; represents the mapping relationship value between the u-th branch of service j and microservice i; b i，n′ The value identifies whether microservice i is deployed on edge server node n". If so, its value is 1. If not, its value is 0; represents the average sojourn time of microservice i in the processing queue of edge server node n"; b i″，n′ The value identifies whether microservice i" is deployed on edge server node n". If so, its value is 1; if not, its value is 0; represents the download rate between the starting edge server node and edge server node n"; β l，i″ The value indicates whether microservice i" includes the l-th layer of mirror data. If so, its value is 1; if not, its value is 0.

[0069] Specifically, based on the above (that is ), it can be known that that is, b i，n The value of is monotonically increasing with respect to the objective function. Considering the objective function of multiple services, multiple nodes, multiple microservices, and multiple branches, analyze b i″，n′ affected by external variables (including other deployments b i，n and ): When is relatively small, that is, when the traffic from other edge server nodes and the traffic forwarded to other edge server nodes is small, is small; When is relatively small, that is, when the sojourn time (queueing delay) of all microservices i" running on edge server node n" is small, is small; When is relatively small, that is, when the change in deployment b i″，n′ has a relatively small impact on the sojourn time of other microservices running on edge server node n", Smaller; When it is smaller, that is, when the download time of deploying microservice i″ on edge server node n″ is smaller, It is smaller.

[0070] Based on the above analysis, assuming all local deployments initially, then And Will be larger and difficult to meet conditions C1, C2, and C5. By changing b i″，n″ = 1 to b i″，n″ = 0, it will force To change, and even The change of, that is, it will reduce the Corresponding to (i″, n″) while increasing the Corresponding to other nodes (i″, n≠n″). When the increase is less than the decrease, it is judged that b i″，n″ = 0, that is, delete this deployment and orchestrate the corresponding service to other better deployment nodes.

[0071] Based on the above analysis, the following is given: Among them, Is the gradient value of the joint optimization objective function with respect to the deployment variables of all edge server nodes for deploying the current microservice i″, Is the corresponding number of deployment nodes, Is the relative gradient value of deploying microservice i″ on node n″ compared to the average deployment on other nodes, and select the deployment pair (i , n * , n * ) with the largest * , n * ) in all deployments as the deployment strategy that needs to be and can be adjusted, where (i * In the edge server node n * . It should be noted that adjusting the deployment strategy potentially requires orchestration adjustment, that is, orchestrating the service deployed on (i * , n * ) to the microservice on other deployment nodes for (i * , n≠n * ). Therefore, Not only represents the need to cancel the deployment corresponding to the larger , but also requires To be smaller, with potential orchestration adjustability.

[0072] 2) Microservice Orchestration Analysis Microservice Orchestration Based on Deployment and Branch will affect the total latency τ for completing the inference task j(n) , that is, selecting the appropriate edge server node and branch will accelerate the microservice operation: Based on the combined optimization objective function of deployment and orchestration, the same analysis can be carried out of the value affected by external variables (such as deployment b i，n , and ): To reflect the influence values of the transmission rate, processing latency, and overall queuing latency of the edge server node, calculate the gradient value of the combined optimization objective of deployment and orchestration for the edge server node orchestration variable of the microservice, then we can obtain: Where: s Among them, represents the gradient value of the optimization objective function for the orchestration variable when orchestrating business j(n) on the microservice i″ deployed on the edge server node n″; represents the corresponding influence value of the transmission rate in this gradient value; represents the corresponding influence value of the processing latency; represents the corresponding influence value of the overall queuing latency; Based on (that is ), the same analysis can be carried out affected by external variables (such as deployment {b i，n}, and ): Influence of: When on the microservice flow of business j(n), microservice i″ and its previous and subsequent microservices are completed on the same edge server node, then Otherwise, when its previous and subsequent microservices are completed on edge server nodes with higher transmission rates, is smaller; Influence of: When the processing latency is smaller when business j(n) is completed on the microservice i″ deployed on node n″, is smaller; Influence of: indicates that when business j(n) is completed on the microservice i″ deployed on node n″, the overall queuing latency will increase; Based on the above analysis, when a certain deployment is updated to it will force then this microservice (i * , n * , n *) The originally arranged service j(n) will need to be readjusted, given: That is, the microservice i on the microservice flow of service j(n) * , is orchestrated to the deployment (i , n * ) with the minimum transmission and computing cost ( ** minimum).

[0073] 3) Business-oriented branch selection The value generated by business-oriented branch selection will affect the calculation accuracy e u and the total latency τ for completing its inference task j(n) . Therefore, to determine the microservice set of service j(n), the issues to be considered are: When adopting the early branch exit mechanism, calculate the gradient value of the joint optimization objective of deployment and orchestration for the DNN branch selection variable, and optimize the branch selection based on this gradient value. Use the following formula to calculate this gradient value Where: Among them, represents the gradient value of the joint optimization objective of deployment and orchestration for the selection variable of the u-th branch of service j(n), represents the transmission time impact value of the u-th branch of service j(n) affected by microservice deployment, orchestration, and the branch together, represents the service queuing time impact value of the u-th branch of service j(n) affected by microservice deployment, orchestration, and the branch together, represents the impact value on the service queuing time of other microservices of the u-th branch of service j(n) affected by microservice deployment, orchestration, and the branch together; represents the accuracy impact value of the u-th branch of service j(n).

[0074] Based on the calculation formula of the gradient value , it can be seen that branch u determines the accuracy issue, while is jointly determined by the branch selection strategy and the deployment strategy {b i，n} and the orchestration strategy , among which, the orchestration strategy is strongly coupled with the branch selection strategy , that is, the downgraded branch will cause Equivalent deletion (deleting microservices on the microservice flow), and even iterative optimization will trigger {b i，n} to be updated and adjusted. Considering the implementability and iterativeness of the algorithm, assume that the current branch of business j(n) is u j(n) , when the branch early exits as 1 ≤ u j(n) < U, and when is not updated, then only through changes to measure the advantages and disadvantages of each branch . When is relatively small, it indicates that is relatively small.

[0075] According to the above three aspects of analysis, microservice deployment is the premise of orchestration, that is, the decision of which microservices to orchestrate in business and which edge server node to orchestrate to strongly depends on the deployment decision {b i，n}, and strongly depends on the branch selection decision At the same time, deployment can be oriented to orchestration and can obtain a better overall result. Based on this, the present invention decouples the above problems into two mutually iterative problems of deployment optimization and microservice orchestration optimization, and respectively adopts algorithms of deployment iterative deletion, orchestration iterative update, and branch degradation.

[0076] As an example, the following algorithm can be used for joint optimization of deployment, orchestration, and branch selection. In this algorithm, the strong coupling between the deployment decision {b i，n} and the server node orchestration is given priority, so that b i，n is eliminated (that is, the deployment of microservices on the edge server node is deleted) to trigger adjustment. In view of the fact that b i，n and may induce fluctuations at the initial stage of iterative elimination of deployment, therefore, a trigger flag parameter: F lag parameter is introduced here. When the joint optimization goal of deployment and orchestration cannot be further optimized only through the search of b i，n and , F lag = 1 can be used to trigger branch degradation to participate in the iterative algorithm to optimize this goal.

[0077] Based on the above algorithms, the specific steps for the joint optimization of deployment and orchestration of the present invention in step S120 are as follows: Determine the joint optimization target of deployment and orchestration based on the startup time of microservice deployment, service latency, and service accuracy. Perform joint optimization including microservice deployment optimization and microservice orchestration optimization according to the joint optimization target of deployment and orchestration. When the result of the joint optimization cannot achieve the joint optimization target of deployment and orchestration, further trigger the orchestration iterative adjustment based on the branch optimization of the multi-branch deep neural network model to achieve the joint optimization target of deployment and orchestration.

[0078] Among them, the microservice deployment optimization includes microservice deployment reduction; the microservice orchestration optimization includes replacing the edge server nodes of the microservice deployment.

[0079] In the embodiment of the present invention, the joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the joint optimization target of deployment and orchestration may include: Calculate the sum of gradients of the joint optimization objective function of deployment and orchestration for all edge server nodes where microservices are deployed, and determine the number of all edge server nodes where the microservices are deployed; Based on the calculated sum of gradients and the determined number, calculate the relative gradient value of deploying a microservice on one edge server node and the average deployment on other edge server nodes. Determine the microservice deployment reduction strategy by selecting the deployment with the largest relative gradient value in all deployments; and The orchestration optimization includes: Based on the microservice deployment reduction strategy, through orchestration update, orchestrate the microservices on the microservice flow of the service to the edge server node deployment with the minimum transmission and computing cost.

[0080] When the result of the joint iterative optimization cannot achieve the joint optimization target of deployment and orchestration, further perform the orchestration iterative adjustment based on the branch optimization of the multi-branch deep neural network model, which may include: When the target value of the joint iterative optimization no longer decreases and satisfies the branch optimization constraint conditions, trigger the branch iterative optimization of the multi-branch deep neural network model, and perform orchestration update after each branch optimization to orchestrate the microservices on the microservice flow of the service to the edge server node deployment with the minimum transmission and computing cost until the joint optimization target of deployment and orchestration is achieved.

[0081] More specifically, the step of calculating the gradient value of the joint optimization objective function of deployment and orchestration for the deployment variables of the current microservice includes: calculating the gradient value of the joint optimization objective function of deployment and orchestration for the deployment variables on each edge server node where the current microservice is deployed And calculate the total gradient value of the deployment variables for all edge server nodes where the current microservice is deployed based on the calculated gradient values to calculate the joint optimization objective function of deployment and orchestration.

[0082] The above-mentioned orchestrating the microservices on the microservice flow of the service to the edge server node deployment with the minimum transmission and computing cost through orchestration updates includes: for the orchestration update decision, calculating the gradient value of the corresponding joint optimization objective of deployment and orchestration for microservice orchestration, selecting the orchestration update decision corresponding to the minimum gradient value to obtain the edge server node with the minimum transmission and computing cost, and orchestrating the microservices on the microservice flow of the service to the selected server node.

[0083] In order to flexibly, quickly, and efficiently deploy AI applications at the network edge, the present invention uses virtualization technology to allocate and manage resources, takes container virtualization as an operating system-level technology, provides an operating system kernel for each container, and enables an independent container engine. Compared with traditional hardware virtualization, containers have the advantages of low resource consumption and fast startup. Therefore, considering the limited edge computing resources and strict latency requirements, adopting container virtualization can improve resource utilization and more efficiently meet the quality of service (QoS) requirements. The present invention introduces microservices based on containers, combines the multi-branch structure of deep neural networks and decomposable microservices, uses multiple independent microservices to complete complex deep neural network inference calculations, and improves the real-time performance and effectiveness of computing through optimizing the microservice deployment on multiple edge computations and the microservice orchestration for completing corresponding tasks under the reasonable allocation and scheduling of limited resources.

[0084] The solution of the present invention realizes the following functions: 1) Deploy microservices on multiple edge servers based on the container image layer, analyze the storage and time costs of the deployment, and the operating efficiency under different deployment degrees; 2) Support DNN applications of AI services, build a multi-branch DNN to meet different service accuracy and latency requirements, and use a directed acyclic graph (DAG) to define the service process of each branch; 3) Based on the deployment scenario, use queuing theory to analyze the computing and sojourn (queuing) time of different branch models under edge collaboration conditions; 4) The present invention proposes a joint optimization problem of microservice deployment, orchestration, and DNN branches, and proposes an effective solution method.

[0085] An embodiment of the present invention also provides a microservice deployment and orchestration optimization system for a deep neural network. The system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0086] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps implemented by the foregoing microservice deployment and orchestration optimization method for a deep neural network are realized. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the technical field.

[0087] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the foregoing microservice deployment and orchestration optimization method for a deep neural network are realized.

[0088] Those of ordinary skill in the art should understand that the various exemplary components, systems, and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Specifically, whether to be implemented in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present invention are programs or code segments used to execute the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave on a transmission medium or a communication link.

[0089] It should be clear that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0090] In the present invention, the features described and / or illustrated for one embodiment can be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with the features of other embodiments or replace the features of other embodiments.

[0091] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for optimizing the microservice deployment and orchestration of a deep neural network, characterized in that, This method is based on a distributed architecture, which includes a cloud center node and edge server nodes communicatively connected to the cloud center node. The method includes: Initial deployment step: The cloud center node deploys multiple microservices composed of different container images to multiple edge server nodes based on the microservice deployment startup time and resource conditions. Among them, each microservice is constructed based on a business branch of a pre-trained multi-branch deep neural network model, and each business branch corresponds to a microservice set. Joint optimization step of deployment and orchestration: Determine the joint optimization goal of deployment and orchestration based on the microservice deployment startup time, service latency, and service accuracy. Perform joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the joint optimization goal of deployment and orchestration. When the result of the joint iterative optimization cannot achieve the joint optimization goal of deployment and orchestration, further perform orchestration iterative adjustment based on the branch optimization of the multi-branch deep neural network model to achieve the joint optimization goal of deployment and orchestration.

2. The microservice deployment and orchestration optimization method of the deep neural network according to claim 1, characterized in that The microservice flow of each business branch is constructed based on a directed acyclic graph; The branch optimization of the multi-branch deep neural network model is a branch optimization algorithm based on the early exit algorithm for branches.

3. The microservice deployment and orchestration optimization method for the deep neural network according to claim 1, characterized in that The deploying multiple microservices composed of different container images to multiple edge server nodes based on the microservice deployment startup time and resource conditions includes: Pull the container images corresponding to each microservice from the set of image layers in the image repository, and download the pulled container images to the matching edge servers according to the pre-set constraint conditions, so as to deploy multiple microservices composed of different container images to multiple edge server nodes; The constraint conditions include: The startup time of the deployed microservice is less than a predetermined image download latency threshold; The remaining computing resources and storage resources of the edge server node where the microservice is to be deployed are greater than the computing resources and storage resources required by the microservice.

4. The method for optimizing microservice deployment and orchestration of a deep neural network according to claim 1, characterized in that, The microservice deployment optimization includes microservice deployment reduction; The microservice orchestration optimization includes replacing the edge server nodes where the microservices are deployed.

5. The microservice deployment and orchestration optimization method for a deep neural network according to claim 1, wherein The joint optimization goal of deployment and orchestration is determined based on the following formula: Among them, F(ψ) represents the joint optimization objective of deployment and orchestration, ψ is the optimization variable, N represents the number of edge server nodes, U represents the number of branches of the deep neural network model, I represents the number of microservices, ω1 is the latency weight parameter of the service, ω2 is the accuracy weight parameter of the service, ω3 is the weight parameter of the deployment latency, and j(n) represents the service arriving at edge server node n; τ j(n) represents the download start time of service j(n), The value of indicates whether to select branch u of the deep neural network model to process service j(n), e u represents the computational accuracy of branch u of the deep neural network model; represents the download start time required for the deployment of microservice i on edge server node n.

6. The method for optimizing microservice deployment and orchestration of a deep neural network according to claim 5, characterized in that The optimization variables are: Among them, {b i，n} represents the deployment decision of the microservice on the edge server node, represents the branch selection decision of the deep neural network model, represents the microservice orchestration decision, The value of indicates whether the microservice i on the edge server node n' processes the service j(n).

7. The method for optimizing microservice deployment and orchestration of a deep neural network according to any one of claims 1 to 6, characterized in that The performing joint iterative optimization including microservice deployment optimization and microservice orchestration optimization according to the joint optimization goal of deployment and orchestration includes: Calculate the gradient value of the joint optimization objective function of deployment and orchestration with respect to the deployment variables of the current microservice; Based on the calculated gradient value and the number of edge server nodes where the current microservice is deployed, calculate the relative gradient value of deploying the current microservice on one edge server node and the average deployment on other edge server nodes. Determine the microservice deployment reduction strategy by selecting the deployment with the largest relative gradient value in all deployments; The joint iterative optimization includes: Based on the microservice deployment reduction strategy, through orchestration update, orchestrate the microservices on the microservice flow of the service to the edge server node deployment with the minimum transmission and computing cost; When the result of the joint iterative optimization cannot achieve the joint optimization goal of deployment and orchestration, further perform an orchestration iterative adjustment based on the branch optimization of the multi-branch deep neural network model, including: When the optimization objective value of the joint iterative optimization no longer decreases and satisfies the branch optimization constraint conditions, trigger the branch iterative optimization of the multi-branch deep neural network model, and perform an orchestration update after each branch optimization, so as to, through the orchestration update, orchestrate the microservices on the microservice flow of the service to the edge server node deployment with the minimum transmission and computing cost, until the joint optimization goal of deployment and orchestration is achieved.

8. The microservice deployment and orchestration optimization method of the deep neural network according to claim 7, wherein: The step of calculating the gradient value of the joint optimization objective function of deployment and orchestration with respect to the deployment variable of the current microservice includes: calculating the gradient value of the joint optimization objective function of deployment and orchestration with respect to the deployment variables of each edge server node for deploying the current microservice, and calculating the total gradient value of the joint optimization objective function of deployment and orchestration with respect to the deployment variables of all edge server nodes for deploying the current microservice based on the calculated gradient values; The step of, through the orchestration update, orchestrating the microservices on the microservice flow of the service to the edge server node deployment with the minimum transmission and computing cost includes: for the orchestration update decision, calculating the gradient value of the corresponding joint optimization objective of deployment and orchestration with respect to the microservice orchestration variable, selecting the orchestration update decision corresponding to the minimum gradient value to obtain the edge server node with the minimum transmission and computing cost, and orchestrating the microservices on the microservice flow of the service to the selected server node.

9. The microservice deployment and orchestration optimization method for a deep neural network according to claim 1, characterized in that In the initial deployment step, the front and rear microservices on the microservice flow are deployed on the nearest edge server nodes, and the longest branch in the multi-branch deep neural network model is used as the currently selected branch.

10. A microservice deployment and orchestration optimization system for a deep neural network, characterized in that, The system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method according to any one of claims 1 to 9.

Images