Distributed online AI service collaborative awareness method for unmanned aerial vehicle cluster

Abstract

The invention provides an unmanned aerial vehicle cluster distributed online AI service collaborative awareness method. The method comprises a data and model joint scheduling decision algorithm based on centralized offline training, a distributed online reasoning framework and deep reinforcement learning. According to the invention, a complex situation awareness task is divided into a target identification-target tracking-target positioning AI service chain; an AI service is cooperatively provided by using an unmanned aerial vehicle cluster; a partially observable Markov decision process is adopted for modeling environment-action-reward of the intelligent agent; and finally, training a decision model based on a deep reinforcement learning algorithm, and deploying the model to each unmanned aerial vehicle node for independent data scheduling and model dynamic activation decision. According to the method, the problem of dynamic coordination of chained AI service tasks (such as situation awareness tasks and micro-services) is effectively solved, higher flexibility and expansibility are achieved, the resource utilization rate is also remarkably improved, and meanwhile, stable operation of the system can still be kept when a node fault occurs.

Description

Technical Field

[0001] The present invention relates to the field of situational awareness tasks and distributed system technology, and proposes a distributed online AI service coordination perception method based on deep reinforcement learning. Background Art

[0002] In today's complex and ever-changing situational awareness tasks, service demands are increasingly diversified and chain-like. These tasks often rely not on a single independent AI service, but rather require multiple AI services to work together in a chain to provide comprehensive and accurate services. However, traditional AI service collaboration methods are insufficient in the face of such complex scenarios. Most of them are tailored for specific scenarios and lack sufficient flexibility and versatility. During the data processing process, the relationship between AI services and nodes is often firmly bound together. Once the service environment changes, the entire system needs to be extensively adjusted and restructured, which undoubtedly increases the maintenance cost and complexity of the system.

[0003] In recent years, with the continuous development of self-learning technology, a number of new AI service coordination methods have emerged. These methods attempt to leverage advanced techniques such as deep reinforcement learning and context-based banding algorithms to learn service coordination strategies from experience and adapt to the ever-changing service environment. However, most of these self-learning methods adopt centralized architectures, facing numerous challenges during implementation in specific environments. First, centralized methods require access to global knowledge, which is particularly difficult in distributed, large-scale network environments. Because network nodes may be geographically distributed, communication latency and bandwidth limitations between them will affect the efficiency of global knowledge acquisition. Second, centralized methods require processing large amounts of data during training and inference, placing high demands on computing resources. In a distributed environment, how to efficiently allocate and utilize computing resources is a pressing issue.

[0004] Therefore, to overcome the aforementioned challenges, a distributed online AI service collaborative perception system and method for drone swarms based on deep reinforcement learning agents has emerged. This method fully leverages the flexibility and distributed nature of drone swarms, distributing the tasks of AI service coordination to individual drone nodes. Each drone node deploys a self-determining DRL (Deep Reinforcement Learning) agent, which uses local observation information to make decisions on data scheduling, routing, and the dynamic activation and migration of AI models. Because each drone node can make online decisions based solely on local observations, the proposed method can efficiently process large amounts of real-time situational data, even in large-scale networks, providing accurate and timely services for situational awareness tasks. Summary of the Invention

[0005] The present invention provides a distributed online AI service collaborative perception system and method, which aims to solve the dynamic coordination problem of chain AI services (such as situational awareness tasks, microservices, etc.). First, in the design stage, an intelligent agent with the ability to execute multiple decisions is designed, including data scheduling, routing, and dynamic activation and migration of AI services, covering multiple tasks, which can better and more efficiently solve service coordination problems. Secondly, in the model training stage, a centralized offline training online distributed reasoning framework is proposed. By combining multi-environment parallel sampling, the convergence of the strategy is accelerated, a unified deep neural network model is trained, and then deployed to each network node, and distributed online reasoning is performed during operation. The agent makes independent decisions based on local observations to achieve lower latency response. Finally, in the testing stage, by unifying the dynamic activation, migration of services with data scheduling and routing, the agent can make independent joint decisions on data and AI services based on local observations, achieving lower latency and more reliable data processing.

[0006] The present invention mainly includes the following aspects:

[0007] Distributed drone cluster architecture. The present invention utilizes the flexibility of drones to collaboratively process situational awareness tasks and proposes a distributed drone cluster architecture. The architecture consists of three types of nodes: entry nodes, processing nodes, and exit nodes. The entry node is responsible for collecting situational data and passing this data to the processing node for processing. The processing node is responsible for receiving this data and, by sensing the resource usage of surrounding neighboring nodes, performing data scheduling and AI service deployment to complete the processing of situational data. The exit node is the final flow of data, that is, after the data processing is completed, the data is finally routed to the exit node for result storage.

[0008] Centralized training and distributed reasoning architecture. The present invention is based on the idea of centralized offline training and distributed online reasoning. The ACKTR algorithm is selected to train the DRL agent using multiple parallel environment copies. The ACKTR algorithm is an extension method of Actor-Critic (A3C). It can obtain more diverse training data through multiple parallel environment copies to train a single global model. After the training converges, this single global model can make decisions for any node in the network, and can also distinguish different nodes according to different observation results and take appropriate actions. Therefore, during the online reasoning process, the DRL agent can be deployed separately on each network node to perform fully distributed online reasoning.

[0009] Design of a Partially Observable Markov Decision Process. The present invention models the problem of collaborative data processing by drone clusters as a partially observable Markov decision process (POMDP). The POMDP includes the partially observable observation space of the drone nodes, the action space, the state transition equation, and the reward function. The state space includes the processing progress of the current data stream, the available resources of neighboring nodes, link usage, and service deployment. The action space includes whether the data is processed locally or dispatched to a neighboring node for processing. The state transition is different depending on the currently selected action. The situation of collaborative data processing by the drone cluster will be different, and the next state will also be different. The reward function is calculated based on the quality of data processing after each decision. By mathematically representing the complex problem of collaborative data processing by drone clusters and customizing the design of the POMDP, the complex real-world problem is transformed into a computable and optimizable model, making it solvable.

[0010] Design of a joint data and model decision-making solution. Due to the limitations of traditional approaches, this paper comprehensively considers data scheduling and routing, as well as model activation and migration, within a distributed architecture when designing an intelligent agent. This design employs an intelligent agent capable of executing multiple decisions simultaneously, enabling a more efficient and effective solution to service collaboration issues.

[0011] The specific technical solutions provided by the present invention are as follows:

[0012] A distributed online AI service collaborative perception method for drone swarms includes the following steps:

[0013] Step 1: Build a distributed drone cluster architecture; determine the number of drone nodes and the type of each drone node in the distributed drone cluster architecture, and divide drone nodes into entry nodes, processing nodes, and exit nodes; and determine the connection relationship between drone nodes in the drone cluster, the size of drone computing resources, and the link capacity between connected drones;

[0014] Step 2: Construct a partially observable Markov decision model for chain processing of data;

[0015] The entry node obtains situation data and transmits it to the processing node. The processing node processes the data based on a partially observable Markov decision model, and the results of the data processing are routed to the exit node for storage. The situation data specifically includes the AI model requested by the current data flow, the entry node and exit node of the current data flow in the drone network, the time when the current data flow arrives at the entry node, the maximum acceptable delay of the current data flow in the drone network, the computational complexity of the current data flow, the data transmission rate required by the current data flow, and the data volume of the current data flow.

[0016] The model is specifically established as follows:

[0017] Observation Space: The observation space of each processing node includes the incoming flow, link information and resource information of the current processing node and its neighboring processing nodes, as well as information about the AI models deployed on the node. The neighboring processing nodes are specifically processing nodes that have a connection relationship with the current processing node. The information in the observation space needs to be standardized as follows:

[0018] The data flow information of the incoming flow includes two flow attributes: the completion progress of the incoming flow Increases to 1 over time; the current remaining time accounts for the percentage of the maximum acceptable delay τ f ∈[0,1], decreases towards 0 over time;

[0019] The link information includes: the resource usage of all outgoing links of the current processing node v The shortest path delay D from the current processing node v through each neighbor node to the exit node of the data flow v,f ∈|-1,1|;

[0020] The resource information includes: the usage of computing resources of the current processing node v and all its neighboring nodes

[0021] The deployed AI model information includes the model indicator variable X v ∈{0,1}, indicating whether the currently requested AI model is available on the current node and its neighboring nodes.

[0022] Action space: Make action decisions based on network degree and observation space, determine the processing nodes for stream processing and process the decisions;

[0023] Reward function: sparse rewards combined with dense rewards are set based on the success or failure of data processing;

[0024] Step 3: Centralized offline training is performed based on the ACKTR algorithm. The ACKTR algorithm is specifically an actor-critic network framework that introduces a Kronecker factorized trust region for gradient optimization training. Specifically, each processing node is trained based on the established partially observable Markov decision model and the ACKTR algorithm. The local decision experience (current observation, action, reward, next observation) is added to the global experience batch. The global model is updated based on the global experience batch parameters, and the updated global model parameters are distributed to each processing node to complete the update of the model parameters on the processing node.

[0025] Step 4: Scheduling decision for AI services;

[0026] The processing node activates and migrates AI services and schedules and routes data based on current observations and action decisions. The action decisions are as follows: First, the same action space A∈{0,1,…,ΔG} is set on each processing node, representing the connection relationship between neighbor nodes, where ΔG is the network degree. If a node has fewer than ΔG neighbors, a virtual node is added. If action A=0, the flow will be processed locally. If action A>0, the flow will be sent to the ath neighbor v of the current processing node v. a ∈V v At the neighbor node v a Services will be provided based on the corresponding AI model.

[0027] Furthermore, the remaining time before the maximum acceptable delay is a percentage τ f Obtained through the following calculation:

[0028]

[0029] Among them, t represents the current time, τ f Indicates the maximum acceptable delay, Represents the time when the current data flow arrives at the entry node;

[0030] Resource usage of all outgoing links of the current processing node v Obtained through the following calculation:

[0031]

[0032] Among them, l represents the current egress link, L v represents all outgoing links, Indicates the transmission rate cap of all outgoing links of the node L The maximum value of r′ l (t) = cap l -r l (t), is the remaining transmission rate of the egress link, rl (t) is the transmission rate of the egress link, λ f is the data rate;

[0033] The shortest path delay D v,f Obtained through the following calculation:

[0034]

[0035] in Denotes the path delay from the current node v to the exit of flow f via a neighbor node v′. If there is no path from the next neighbor node to the exit node, D v,f Set to -1 to indicate that the path does not exist;

[0036] The usage of computing resources of the current processing node v and all its neighboring nodes Obtained through the following calculation:

[0037]

[0038] in Represents the maximum value of the computing power resources of the node and all neighboring nodes, r′ v (t) = cap v -d f , is the remaining computing power resources of the processing node, d f Indicates the computational complexity of the current data flow.

[0039] The model indicator variable X v =x v,i (t)|v∈V v , where x v,i Represents AI model m i Is it placed on node v at time t, i is the number of the deployed AI model; when x v,i When (t) = 0, it means that the processing node does not have any AI model at this time. v,i When (t)=1, it means that the processing node currently has the i-th AI model.

[0040] The present invention is aimed at the problem of data collaborative processing in situational awareness tasks, and proposes a reliable and efficient distributed drone cluster online AI service collaborative perception system and method based on deep reinforcement learning. Specifically, for complex and changeable network environments, this method has higher flexibility and scalability, and can dynamically adjust the services of nodes according to the needs of the task to adapt to the ever-changing service environment. At the same time, this method can also make full use of the computing resources of drone nodes to achieve efficient training and reasoning. Therefore, the drone cluster distributed online AI service collaborative method based on deep reinforcement learning intelligent agents not only solves the defects in the current AI service coordination method, but also provides new ideas and solutions for situational awareness tasks, which is an important supplement and improvement to existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Represents the drone swarm architecture;

[0042] Figure 2 Represents the data processing process;

[0043] Figure 3 Represents a centralized offline training and distributed online inference framework;

[0044] Figure 4 Shows a flow chart of Example 1;

[0045] Figure 5 This shows a signaling diagram of Example 1;

[0046] Figure 6 Shows the flow chart of Example 2;

[0047] Figure 7 This shows a signaling diagram of Example 2;

[0048] Figure 8 Shows a flow chart of Example 3;

[0049] Figure 9 This is a signaling diagram of Example 3; DETAILED DESCRIPTION

[0050] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of the distributed online AI service collaborative perception method of a drone cluster in conjunction with the accompanying drawings.

[0051] A distributed online AI service collaborative perception method for drone swarms includes the following steps:

[0052] Step 1: Build a distributed drone cluster architecture; Figure 1As shown in the figure, the number of drone nodes in the distributed drone cluster architecture is determined. Subsequently, the type of each drone node is determined, namely, whether it is an entry node, a processing node, or an exit node. Different nodes have different responsibilities. For example, the entry node is responsible for collecting situational data and transmitting it to the processing node for processing. The processing node receives this data and, by sensing the resource usage of its connected neighboring nodes, performs data scheduling and AI service deployment to complete the situational data processing. The exit node is the final destination of the data. After processing, the data is finally routed to the exit node for storage. Next, the connections between the drones in the drone cluster are determined. Since data needs to be transmitted between nodes, it is necessary to determine which nodes have connections to facilitate message and data transmission. Finally, the computing resource size of each drone in the drone cluster and the link capacity between connected drones are determined. This is to simulate the different resources on nodes in a real environment and to enable the intelligent agent to maximize resource utilization when making decisions.

[0053] like Figure 2 As shown, the distributed UAV cluster architecture designed by the present invention can achieve: data collection, the original situation data needs to be collected from various places in the unknown area, and the entry node is responsible for collecting these data; data processing, in the distributed UAV cluster, the data is chain-processed through data scheduling, routing, and model activation and migration; data storage, by defining the exit node, the results of data processing can be finally routed to a specific node for storage.

[0054] Step 2: Construct a partially observable Markov decision process model to perform chain processing of data;

[0055] How to solve chained data processing is a key issue in situational awareness. In a distributed drone swarm, the deployment of services and the use of related resources vary across each drone node, making it difficult for multiple drones to collaboratively complete chained data processing. Therefore, this paper addresses this problem mathematically by using a partially observable Markov decision process (POMDP). Specifically, this POMDP addresses: 1) Local observation limitations in distributed environments: In distributed service coordination scenarios, each node only has access to local information about itself and its immediate neighbors, and lacks global network awareness. 2) Formalizing complex dynamic problems: Service coordination involves the dynamic activation and migration of AI services, as well as the joint optimization of data scheduling and routing. POMDP transforms this problem into a sequential decision-making task using reinforcement learning. Through the mathematical definition of state, action, and reward, the agent learns strategies to adapt to dynamic environments from experience. 3) Addressing uncertainty and dynamic changes: Data patterns, resource loads, and topology in real-world network environments can change rapidly and are difficult to fully predict. POMDP uses probabilistic state transitions and reward mechanisms to help agents optimize long-term benefits amidst uncertainty. The most important thing is how to design the observation space, action space and reward function in POMDP. Based on the original problem, this paper customizes them as follows:

[0056] Observation space design: This includes node resources, link utilization, and the shortest path delay to the exit node. Dummy nodes are used, meaning "-1" is used as the fill value. For neighboring nodes or links that do not exist in the node observation, their corresponding observation values (such as link utilization and neighboring node utilization) are set to "-1." This ensures that the observation vectors for each node have the same dimensions, helping to avoid decision errors caused by missing values. Normalization also involves normalizing each observation value based on its maximum value, which helps to keep the observations within a similar numerical range, ensuring input consistency, accelerating the convergence of the neural network, and improving learning efficiency.

[0057] Specifically, the observation space can be represented by a five-tuple. The observation space of each agent is limited to information about the incoming flow f, the current processing node v, and its neighboring nodes. These observations are normalized to the range [-1, 1]. Ensuring that all observations are in a similar range is important for the effective training and generalization of deep neural networks. Each part is described in detail below:

[0058] F f Indicates data flow information, It consists of two related stream attributes: Indicates the completion progress of data stream f. The completion progress of a data stream is based on the linear structure of the input data stream, the target identification-target tracking-target positioning defined in this article, and the portion of data currently being processed by the node. For example, if the node is currently processing the target tracking data, the processing progress of the data stream is 2 / 3 and increases towards 1 over time. and where τ f Indicates the maximum acceptable delay of the task flow, t represents the current time, Represents the time when the current data flow arrives at the entry node; so Represents the remaining time of the task flow relative to the maximum acceptable delay. It indicates the percentage of the current remaining time to the maximum acceptable delay, and decreases towards 0 over time.

[0059] Indicates the resource usage of all outgoing links of node v, Includes the resource usage of all outgoing links of node v. Where l represents the current outgoing link, L v Indicates all outgoing links; Indicates the maximum transmission rate of all outgoing links of the node, through which Normalized. And r′ l (t) = cap l -r l (t), is the remaining transmission rate of the egress link, r l (t) is the transmission rate of the outbound link, minus the data rate λ f , because when r′ l (t)-λ f When ≥0, the link can continue to forward data flow.

[0060] Indicates the usage of computing resources of node v and all its neighboring nodes, in Indicates the maximum value of the computing power resources of the node and all neighboring nodes, through which Normalized. And r′ v (t) = cap v -d f , is the remaining computing power resources of the node, d f Indicates the computational complexity of the current data stream, minus d f , because when r′ v (t)-d f ≥0, the node can continue to process the data stream.

[0061] d v,f represents the shortest path delay, in represents the path delay from the current node v to the exit of flow f via a neighboring node v′. Then D v,f is the shortest path delay from the current node v through each neighbor to the exit of flow f, which is equal to the remaining time of flow f If there is no path from the next neighbor node to the exit node, then d v,f Set to -1 to indicate that the path does not exist.

[0062] X v : Model indicator variable X v = <x v,i (t)|v∈V v >∈{0,1} is used to indicate whether the currently requested AI model is available on the current node and its neighboring nodes. v,i Represents AI model m i Is it placed on node v at time t, i is the number of the deployed AI model; when x v,i When (t) = 0, it means that the processing node does not have any AI model at this time. v,i When (t)=1, it means that the processing node currently has the i-th AI model.

[0063] Action space design: Action space A∈{0,1,…,ΔG}, whenever a flow f arrives at a processing node, the DRL agent deployed on the node will take action, and the action space {0,1,…,ΔG} of all processing nodes is the same. If a node has fewer than ΔG neighbors, a virtual node is added. According to the network degree ΔG, the DRL agent at node v performs the relevant operation A∈{0,1,…,ΔG}, that is, specifies how to set If action A = 0, the flow will be processed locally, that is, When action A>0, the flow will be sent to the ath neighbor v of node v. a ∈V v and dispatch the data to the neighboring node v a When v a The corresponding model m will be dynamically activated i This is how we provide services.

[0064] Reward function design: By combining sparse rewards (+10 for successfully processing a piece of data, -10 for failure) with dense rewards (such as +1 / ns for processing part of the data in a chain), we can balance exploration and exploitation and accelerate strategy convergence.

[0065] Step 3: Centralized offline training based on the ACKTR algorithm; POMDP obtains an agent that can select the optimal action based on local observations by modeling local observations and combining them with centralized offline training. This allows the agent to map different local observations to the optimal action, thus allowing the agent to make decisions based only on local observations without relying on global knowledge.

[0066] The ACKTR algorithm is an extended version of A3C and is a distributed reinforcement learning algorithm under the AC (Actor-Critic) framework. The actor-critic network framework consists of two main parts: the policy network, which plays the role of the performer. It is responsible for displaying the output based on the environment state input s (that is, the established observation space) and making action decisions A. The second is the Q network, which is the critic. After the performer performs, it will score the performer's actions and evaluate the value of the action Q. w (s, a). The performer updates its network parameters θ based on the critic's scores, striving for more accurate and precise next actions. The critic, in turn, updates its own network parameters w based on the reward function feedback, striving for more accurate scores. By constraining the value of the performer's actions, the performer's learning update direction is improved. Building on this foundation, the ACKTR algorithm introduces the Kronecker Factorized Trust Region (K-FAC) optimizer to effectively approximate the natural gradient. It also supports synchronous and parallel environment sampling, achieving more diverse samples to improve the update stability and convergence speed of the strategy.

[0067] like Figure 3 As shown in the figure, first, through centralized global model training, multiple parallel environment copies (referring to multiple independent but identically configured network environment instances) run in parallel at the same time to collect diverse training data. This can accelerate the training process and improve the generalization ability of the model, and finally generate a unified global policy model; then, an independent DRL agent is deployed on each scheduling node, and the trained global model is copied to the DRL agent on each processing node (the DRL agent is essentially the instantiation of the global model on each network node). The DRL agent of each node makes decisions based on local observations to achieve distributed online reasoning.

[0068] Step 4: Scheduling decision for AI services;

[0069] The processing node state defined in this invention has the ability to execute multiple decisions, including the activation and migration of AI services, and the scheduling and routing of data. Specifically, this is achieved by designing two main decision variables, as shown below:

[0070] First, the first decision variable is the model indicator variable xv,i (t)∈{0,1}, i={1,2,3} to indicate three AI models m i Is AI model m placed on node v at time t? i This includes target recognition, target tracking, and target positioning. When the variable is 0, it means that no service is available on the node at that time, that is, no AI model is activated. A variable of 1 indicates that an AI model is currently activated on the node. At the same time, an AI model can be dynamically activated on multiple nodes simultaneously, meaning that an AI model can exist on multiple drone nodes at the same time. However, each node can only activate a maximum of one AI model to provide services at any given time.

[0071] In addition, how data is scheduled between nodes introduces another decision variable, the data scheduling variable To indicate how to schedule and route, where V v is the set of all neighbor nodes of node v. Specifically, when When t indicates that the data flow f requires the activation of the relevant model m on the processing node v at time t i Processing the data requires x v,i (t) = 1, that is, the node v has AI model m at this time i . And when When t = t, it means that the data flow f is not processed locally at time t, but needs to be scheduled to the neighboring node v′ along the link l = (v, v′) for processing.

[0072] Example 1: Joint decision-making of data scheduling, routing, and model dynamic activation.

[0073] like Figure 4 and Figure 5 As shown, consider the following scenario. In the situational awareness task, data processing is divided into a chain of AI services: target recognition, target tracking, and target positioning. At the same time, these three AI services are pre-placed on all processing nodes, waiting for data to arrive and dynamically activating one of the AI models to provide services based on this, thereby performing online AI service collaboration.

[0074] In the present invention, a distributed UAV cluster architecture is used to collaboratively process data. The UAVs are randomly divided into three types of nodes, namely entry nodes, processing nodes, and exit nodes. The network topology is set for the UAV cluster. Figure 1 As shown in the figure, the entry node is responsible for collecting situation data and transferring the situation data to the The data is transmitted to the drone processing node for data processing. The data is dispatched on multiple processing nodes and the processing nodes dynamically activate AI services to collaboratively process the data. Finally, the data is routed to the exit node within a certain period of time, which is considered to be successful data processing. Among them, the present invention models the collected situation data into the following form, specifically an octet Among them, F is all the data currently input into the drone network, and the specific definition of each attribute of each data is as follows:

[0075] M f ={m i}, i = {1, 2, 3} represents the AI model requested by the current data stream. and They represent the entry node and exit node of the current data flow in the UAV network respectively; Indicates the time when the current data flow arrives at the entry node; τ f It represents the maximum acceptable delay of the current data flow in the UAV network, that is, the time taken for the data to enter the entry node and leave the exit node must not exceed τ f ;d f Indicates the computational complexity of the current data stream, that is, the total amount of computation required to process the data; f Indicates the data transmission rate required by the current data flow; Size f Indicates the size of the current data stream.

[0076] In the present invention, how to efficiently schedule data between nodes. This embodiment provides a centralized offline training and distributed online reasoning framework for model training and decision making.

[0077] The centralized offline training process is as follows:

[0078] First, whenever a data stream arrives at a processing node, due to the local observation o of each node t The observation space of the drone node is modeled by a partially observable Markov decision process, which includes the node capacity of itself and its neighboring nodes, the usage of link capacity, and the processing of situation data. Through operations such as filling and normalization, a unified local observation information of the intelligent agent is formed. t .

[0079] Processing Node The global policy model deployed in this processing node forms actions based on partially observed state information, including local data scheduling and joint decision-making solutions for dynamic activation of AI services.

[0080] Finally, the processing node executes action a according to the decision solution t, the data scheduling and AI model activation information is transmitted to a neighboring node, and the neighboring node activates the relevant AI model according to the message to provide services to process the data. And the data processing status, including scheduling delay, processing delay and other messages are sent to the processing node, which completes the action reward value r t Calculation.

[0081] In summary, at each time step, whenever a data stream arrives at a processing node, the global policy model deployed on the processing node obtains the current local observation o from the environment. t and the previous reward r t-1 , and choose an action a t , multiple processing nodes will use their local decision experience (o t-1 , a t-1 , r t-1 , o t ) are added to the experience batch. Once the batch is full, the relevant parameters in the global policy model are updated. The global policy model will use these experiences to update its parameters, thus completing the centralized offline training of the global model.

[0082] The distributed online reasoning process is as follows:

[0083] The trained global strategy model is then copied to each processing node to complete the distributed deployment of the model. Due to the above training, the global model is very robust and can complete the joint decision-making of optimal data scheduling and dynamic activation of the model, and can schedule data to the optimal processing node for data processing.

[0084] Example 2: Data scheduling decision.

[0085] like Figure 6 and Figure 7 As shown, consider the following scenario: in the situational awareness task, data processing is divided into a chain AI service chain of target recognition-target tracking-target positioning. At the same time, the deployment relationship between drone nodes and AI services is fixed, that is, only one AI model is pre-deployed on each node, so the service provided by each node is fixed, so as to achieve online AI service collaboration.

[0086] In the present invention, a distributed UAV cluster architecture is used to collaboratively process data. The UAVs are randomly divided into three types of nodes, namely entry nodes, processing nodes, and exit nodes. The network topology is set for the UAV cluster. Figure 1As shown in the figure, the entry node is responsible for collecting situational data and passing it to the drone processing node for data processing. Data is dispatched across multiple processing nodes, and the processing nodes dynamically activate AI services to collaboratively process the data. Finally, data processing is considered successful if the data is routed to the exit node within a certain period of time. The situational data collected in this embodiment is specifically the same as in Example 1.

[0087] This embodiment provides a centralized offline training and distributed online reasoning framework for model training and decision making. The centralized offline training process is as follows:

[0088] First, whenever a data stream arrives at a processing node, due to the local observation o of each node t The observation space of the drone node is modeled by a partially observable Markov decision process, which includes the node capacity of itself and its neighboring nodes, the usage of link capacity, and data processing information. Through operations such as filling and normalization, a unified local observation information of the intelligent agent is formed. t .

[0089] Next, the processing node converts the local observation information o t The data is sent to the local agent (the local agent is the DRL agent deployed on the processing node). The local agent forms a joint decision-making plan for local data scheduling and dynamic activation of AI services through the partial observed status information of the processing node.

[0090] Finally, the processing node executes action a according to the decision solution t Since AI services have been deployed on processing nodes in advance and cannot be changed, it is only necessary to pass the data scheduling information to the appropriate neighboring node. For example, if the target recognition processing of the data is completed at the previous node, then the appropriate neighboring node is the node that needs to have the AI service of target tracking, so that the chain processing of the data can be completed. And the data processing status, including scheduling delay, processing delay and other messages are sent to the processing node, which completes the reward value r for the action. t Calculation.

[0091] In summary, at each time step, whenever a data stream arrives at a processing node, the local agent deployed on the processing node obtains the current local observation o from the environment. t and the previous reward r t-1 , and choose an action a t , after which the local agent will experience (o t-1 , a t-1 , r t-1 , o t) is added to the experience batch b. Once the experience batch b is full, the critic network in the global model Actor Network π θ These experiences will be used to update parameters to complete the centralized offline training of the global model.

[0092] The distributed online inference process is as follows: the trained global model is copied to the local intelligent agent on each processing node to complete the distributed deployment of the model. Due to the above training, the global model is very robust and can make the best data scheduling decisions and schedule data to the optimal processing node for data processing.

[0093] Example 3: Joint decision-making for data scheduling and model migration

[0094] like Figure 8 and Figure 9 As shown, consider the following scenario: in the situational awareness task, data processing is divided into a chain of AI services: target recognition, target tracking, and target positioning. At the same time, the deployment relationship between drone nodes and AI services is fixed, that is, only one AI model is pre-deployed on each node, so the service provided by each node is fixed, so as to perform online AI service collaborative perception.

[0095] In this embodiment, a distributed drone cluster architecture is used to collaboratively process data. Therefore, drones are randomly divided into three types of nodes, namely entry nodes, processing nodes, and exit nodes. A network topology is set for the drone cluster. Figure 1 As shown. The entry node is responsible for collecting situational data and passing it to the drone processing node for data processing. Data is dispatched across multiple processing nodes, and the processing nodes dynamically activate AI services to collaboratively process the data. Finally, data processing is considered successful if the data is routed to the exit node within a certain period of time. The situational data collected in this embodiment is specifically the same as in Example 1.

[0096] In this invention, how to efficiently schedule data between nodes. This invention provides a centralized offline training and distributed online reasoning framework for model training and decision making.

[0097] The centralized offline training process is as follows: the processing node receives situation data and perceives the resource usage and service deployment of all its neighboring nodes. It then forms a unified local observation information of the intelligent agent through operations such as filling and normalization by combining its own situation and data processing information. t If a neighbor node with a related service fails to send a message, its corresponding status is set to -1, indicating that it cannot be used.

[0098] Then, the processing node sends the local observation information to the local agent, and the local agent forms a local data scheduling decision plan based on the partial observation status information of the processing node.

[0099] Finally, the processing node executes action a according to the decision solution t , passing the data scheduling and model migration information to a neighboring node, so that the neighboring node receives the data and the model migrated from elsewhere according to the message, and provides corresponding services to process the data. And the data processing status, including scheduling delay, processing delay and other messages, are sent to the processing node, which completes the reward value r for the action. t Calculation.

[0100] In summary, at each time step, whenever a data stream arrives at a processing node, the local agent obtains the current local observation o from the environment. t and the previous reward r t-1 , and choose an action a t , after which the local agent will experience (o t-1 , a t-1 , r t-1 , o t ) is added to the experience batch b. Once the experience batch b is full, the critic network in the global model Actor Network π θ These experiences will be used to update parameters to complete the centralized offline training of the global model.

[0101] The distributed online inference process is as follows: the trained global model is copied to the local intelligent agent on each processing node to complete the distributed deployment of the model. Due to the above training, the global model is very robust and can complete the optimal data scheduling and model migration decisions, and can schedule data to the optimal processing node for data processing.

[0102] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A distributed online AI service collaborative perception method for drone clusters, characterized by: The following steps are involved: Step 1: Build a distributed drone cluster architecture; Determine the number of drone nodes and the type of each drone node in the distributed drone cluster architecture, and classify drone nodes into entry nodes, processing nodes, and exit nodes; and determine the connection relationships between drone nodes in the drone cluster, the size of drone computing resources, and the link capacity between connected drones; Step 2: Construct a partially observable Markov decision model for chain processing of data; The entry node obtains situation data and transmits it to the processing node. The processing node processes the data based on the partially observable Markov decision model. The processed data results are routed to the exit node for storage. The model is specifically established as follows: Observation space: The observation space of each processing node includes the incoming flow, the link information and resource information of the current processing node and its neighboring processing nodes, and the AI model information deployed on the node; the neighboring processing nodes are specifically the processing nodes that have a connection relationship with the current processing node; Action space: Make action decisions based on network degree and observation space, determine the processing nodes for stream processing and process the decisions; Reward function: sparse rewards combined with dense rewards are set based on the success or failure of data processing; Step 3: Centralized offline training is performed based on the ACKTR algorithm. The ACKTR algorithm is specifically an actor-critic network framework that introduces a Kronecker factorized trust region for gradient optimization training. Specifically, each processing node is trained based on the established partially observable Markov decision model and the ACKTR algorithm. The local decision experience (current observation, action, reward, next observation) is added to the global experience batch. The global model is updated based on the global experience batch parameters, and the updated global model parameters are distributed to each processing node to complete the update of the model parameters on the processing node. Step 4: Scheduling decision for AI services; The processing nodes activate and migrate AI services and schedule and route data based on current observations and action decisions.

2. The method for collaborative perception of distributed online AI services in drone clusters according to claim 1 is characterized in that: The situation data specifically includes the AI model requested by the current data flow, the entry node and exit node of the current data flow in the drone network, the time when the current data flow arrives at the entry node, the maximum acceptable delay of the current data flow in the drone network, the computational complexity of the current data flow, the data transmission rate required by the current data flow, and the data volume of the current data flow.

3. The method for collaborative perception of distributed online AI services of drone swarms according to claim 2, characterized in that: The information in the observation space needs to be standardized, as follows: The data flow information of the incoming flow includes two flow attributes: the completion progress of the incoming flow Increases to 1 over time; the current remaining time accounts for the percentage of the maximum acceptable delay τ f ∈[0,1], decreases towards 0 over time; The link information includes: the resource usage of all outgoing links of the current processing node v The shortest path delay D from the current processing node v through each neighbor node to the exit node of the data flow v,f ∈|-1,1|; The resource information includes: the usage of computing resources of the current processing node v and all its neighboring nodes The deployed AI model information includes the model indicator variable X v ∈{0,1}, indicating whether the currently requested AI model is available on the current node and its neighboring nodes.

4. The method for collaborative perception of distributed online AI services in drone swarms according to claim 3, characterized in that: The action decision is as follows: First, the same action space A∈{0,1,…,ΔG} is set on each processing node, representing the connection relationship of neighbor nodes, where ΔG is the network degree. If a node has fewer neighbors than ΔG, a virtual node is added. If action A=0, the flow will be processed locally. If action A>0, the flow will be sent to the ath neighbor v of the current processing node v. a ∈V v At the neighbor node v a Services will be provided based on the corresponding AI model.

5. The method for collaborative perception of distributed online AI services of drone clusters according to claim 4 is characterized in that: The remaining time before the percentage of the maximum acceptable delay τ f Obtained through the following calculation: Among them, t represents the current time, τ f Indicates the maximum acceptable delay, Represents the time when the current data flow arrives at the entry node; Resource usage of all outgoing links of the current processing node v Obtained through the following calculation: Among them, l represents the current egress link, L v represents all outgoing links, Indicates the transmission rate cap of all outgoing links of the node L The maximum value of r′ l (t) = cap l -r l (t), is the remaining transmission rate of the egress link, r l (t) is the transmission rate of the egress link, λ f is the data rate; The shortest path delay D v,f Obtained through the following calculation: in Indicates starting from the current node v through a neighboring node v ′ The path delay to the exit of flow f. If there is no path to the exit node from the next neighbor node, D v,f Set to -1 to indicate that the path does not exist; The usage of computing resources of the current processing node v and all its neighboring nodes Obtained through the following calculation: in Represents the maximum value of the computing power resources of the node and all neighboring nodes, r′ v (t) = cap v -d f , is the remaining computing power resources of the processing node, d f Indicates the computational complexity of the current data flow. The model indicator variable X v =x v,i (t)|v∈V v , where x v,i Represents AI model m i Is it placed on node v at time t, i is the number of the deployed AI model; when x v,i When (t) = 0, it means that the processing node does not have any AI model at this time. v,i When (t)=1, it means that the processing node currently has the i-th AI model.

Images