Intelligent agricultural network assisted by multi-agent reinforcement learning

Abstract

A drone (UAV) agent is provided for forming a two-layer hybrid smart agriculture network architecture. The UAV agent is configured to support two communication protocols: a short-range communication protocol and a long-range communication protocol. Agricultural sensors collect agricultural data such as soil temperature and participate in a first-layer short-range wireless communication network to transmit the collected data to at least one UAV agent. The UAV agent participates in one or more first-layer short-range communication networks to pick up sensor data from one or more clusters of agricultural sensors, and also participates in a second-layer long-range communication network to route the collected sensor data to at least one cloud server. The tasks in both layers are formulated as optimization problems to achieve optimal data delivery and are solved using multi-agent reinforcement learning (MARL) implemented by the Focused Coordinated Multi-Agent Deep Deterministic Policy Gradient (FC-MADDPG) algorithm of this invention.

Description

Technical Field

[0001] The present invention generally relates to node apparatus or methods for transmitting data over a wireless communication network, and more specifically, to transmitting data from agricultural sensors to a remote cloud server in the absence of conventional communication infrastructure. Background Technology

[0002] In an era of explosive global population growth, smart agriculture practices have become essential. The Internet of Things (IoT) paradigm already plays a vital role in many urban applications such as smart utilities and smart cities. However, the application of IoT in agriculture has not been well studied. In fact, IoT is well-suited for many use cases in agriculture, including greenhouse automation, soil monitoring, and predictive analytics for smart agriculture. IoT can bring numerous benefits to agriculture, such as better decision-making using data, risk reduction, cost management, increased business efficiency, and most importantly, the realization of smart agriculture.

[0003] Smart agriculture is mostly used to describe the application of IoT technology in agriculture. For example, by using smart agriculture sensors to monitor the condition of crops, farms can accurately determine how much fertilizer and pesticide to use for optimal efficiency. However, there are still many challenges to realizing smart agriculture. (1) Lack of infrastructure is a major problem in smart agriculture. Farms need infrastructure such as communication facilities and data processing systems to build an architecture to realize smart farms. However, such infrastructure is not widely available. (2) Data transmission and data analysis are challenging. Farms rely on sparsely distributed sensor networks to collect data. However, sensors do not have sufficient resources to process the data. Therefore, the collected data must be delivered to processing centers (such as more powerful devices or cloud servers) for processing and data utilization. (3) Agricultural sensors have limited memory to store large amounts of data. Therefore, it is necessary to transmit the data collected by the sensors to avoid memory overflow. In the absence of the necessary communication infrastructure, data transmission becomes another major problem. (4) In addition, even if a communication infrastructure is established, poor communication connectivity is another problem. Farms are often located in remote areas where communication connectivity may not be strong enough to facilitate fast data transmission. Furthermore, communication links may be blocked by crops and other physical barriers. Communication connectivity should be reliable enough to withstand harsh weather conditions and ensure non-destructive operation. (5) Sensor cost is also an issue. Agricultural sensors are not mass-produced. The specific agricultural sensors to be deployed depend on the type of information the farm wants to collect and the purpose of the solution. A set of sensors covering a farm can cost thousands of dollars. Other challenges also exist. Therefore, many problems need to be solved to make smart agriculture a reality. In conclusion, intelligent and economical agricultural sensors must be manufactured, a data processing platform is necessary, and a communication network must be developed as a bridge to connect the sensors and the platform.

[0004] Therefore, smart agriculture technologies can be divided into four categories: (1) sensor technology, (2) networking technology, and (3) platform technology. Sensor technology focuses on sensor capabilities, such as what communication interfaces are supported, how data is collected, when data is collected, and what data is collected. Networking technology focuses on how to communicate, when to communicate, and how to organize sensors and communication infrastructure into a network to transmit sensor data to data processing centers (such as cloud servers). Because data itself is of little use if farms cannot utilize it, platform technology focuses on how to process data and gain useful insights to make informed decisions.

[0005] Regarding sensor technology, there are existing technologies developing smart agriculture sensor technologies. For example, SoilTech is a joint research project of several US universities to develop sensor systems capable of measuring dynamic variables in managed and unmanaged soils in situ and remotely.

[0006] In terms of platform technology, there are existing technologies under development, such as Microsoft FarmBeats, which utilizes idle TV frequencies to transmit information in order to overcome the challenge of poor communication connectivity.

[0007] However, due to the lack of communication infrastructure, networking technologies serving as a bridge between sensors and platforms are rarely studied. Agricultural network infrastructure is not yet widely available. Therefore, this factor must be considered in the development of networking technologies. Connecting agricultural sensors with wires is impractical. Therefore, sensors communicate using wireless communication interfaces. Different types of wireless communication technologies exist; for example, LTE / 5G and LoRa are long-range wireless communication technologies. However, using LTE / 5G technology in licensed frequency bands for agricultural sensors is expensive. LoRa can operate in unlicensed frequency bands, but it is not a standardized technology and has a very low data transmission rate. In addition, unregulated channel access can lead to severe interference. Therefore, practical communication technologies are short-range communication protocols that are expected to operate in unlicensed frequency bands, such as IEEE 802.11 (Wi-Fi) and IEEE 802.15.4 (Wi-SUN). The maximum communication distance of these short-range communication protocols is 1 km, which is insufficient for agricultural sensors to send data to remote data processing centers. Therefore, a delivery network is needed. However, the communication infrastructure for agricultural sensors deployed in rural areas is not yet available. Therefore, the development of an innovative solution is desired.

[0008] Therefore, it is desirable to provide a practical network architecture for building smart agriculture systems that transmits agricultural sensor data to a data processing center, where it is processed and useful insights are gained to enable farms to make optimal decisions. Summary of the Invention

[0009] Some embodiments of the present invention provide a two-layer hybrid network architecture for smart agriculture, which eliminates the need for conventional communication infrastructure. The provided agricultural network architecture consists of fixed agricultural sensors, unmanned aerial vehicles (UAVs), and a cloud server, where the UAVs act as conventional communication infrastructure to deliver data from the agricultural sensors to the cloud server. For this purpose, the agricultural sensors and UAVs form a first-layer network to transmit sensor data from the sensors to the UAVs. The UAVs and the cloud server form a second-layer network to route sensor data from the UAVs to the cloud server. The agricultural sensors and UAVs communicate using short-range communication protocols. Conversely, the UAVs and the cloud server communicate using long-range communication protocols. Multi-agent reinforcement learning (MARL) techniques are provided in this agricultural network architecture for optimal sensor data delivery. In the provided agricultural network architecture, the UAVs are configured to operate collaboratively. To conserve energy, the UAVs operate only when data transmission is required.

[0010] In some cases, agricultural sensors can form smart agriculture networks, where they are used to monitor a variety of environmental and soil conditions, enabling farmers to optimize crop production, reduce waste, and make informed decisions. These networks consist of a wide range of sensors, each designed to measure specific data. Different types of agricultural sensors exist, including soil moisture sensors, temperature and humidity sensors, pH sensors, electrical conductivity (EC) sensors, nutrient sensors, CO2 sensors, light sensors, and weather stations. Each sensor can be configured to transmit its readings via signals that can be analog or digital. Analog sensors generate continuous electrical signals that are directly correlated with the measured data, while digital sensors typically use protocols such as I2C, SPI, or wireless communication technologies (such as IEEE 802.11, IEEE 802.15.4, LoRa, or LTE / 5G) to convert measurements into digital signals for data transmission across the agricultural sensor network. Integrating these sensors into cohesive agricultural networks allows for the collection and analysis of data in real-time or near real-time, facilitating precision agricultural practices that lead to more sustainable and efficient agricultural operations.

[0011] For example, soil moisture sensors are configured to measure the water content in the soil to help manage irrigation more efficiently. These sensors can use capacitance or resistance to measure moisture levels, sending an electrical signal associated with the amount of water present. Temperature and humidity sensors are configured to monitor air and soil temperature and air humidity levels. These conditions affect plant growth and disease occurrence. Sensors often use thermistors or digital signal outputs to measure temperature and capacitive elements to measure humidity. pH sensors are configured to measure the acidity or alkalinity of the soil, which is crucial for nutrient availability for plants. pH sensors typically use potentiometry, which generates a voltage that varies with pH levels. Electrical conductivity (EC) sensors are configured to assess soil salinity, which can affect plant growth and water uptake. These sensors use electrodes to measure the soil's electrical conductivity, indicating salt concentration. Nutrient sensors are configured to detect the levels of specific nutrients, such as nitrogen, phosphorus, and potassium, in the soil. These sensors can use various methods, including ion-selective electrodes, to generate voltage or current changes in response to the concentration of a specific nutrient. CO2 sensors are configured to measure the concentration of carbon dioxide in greenhouse or field environments. High levels of CO2 can promote plant growth. These sensors typically use infrared gas analyzers or chemical sensors to convert CO2 concentration into an electrical signal. Light sensors (e.g., photosynthetic active radiation sensors) are configured to measure the intensity of light available for photosynthesis. They can detect the wavelengths of light that plants use for growth, typically using photodiodes or phototransistors that convert light into electrical signals. Weather stations are not single sensors; they integrate multiple sensing elements to monitor weather conditions such as wind speed and direction, rainfall, solar radiation, and atmospheric pressure. These stations use a variety of sensing technologies to collect data, which is then used to make informed decisions about planting, irrigation, and harvesting.

[0012] Some embodiments of this invention are based on the understanding that IoT technology has already played a significant role in many urban applications, such as smart utilities, smart cities, infrastructure monitoring, and smart buildings. However, the application of IoT technology in rural agriculture has not been well studied. IoT can bring many benefits to agriculture, enabling smart agriculture. Therefore, there is a need to develop practical IoT technologies to make smart agriculture a reality.

[0013] Some embodiments of the present invention are based on the following understanding: smart agriculture technologies can be divided into three categories: (1) sensor technology, (2) networking technology, and (3) platform technology. Existing technologies for sensor technology and platform technology are under development. However, networking technology, which is a crucial bridge connecting agricultural sensors and cloud servers, has not been studied. Therefore, it is hoped that IoT network technology will be developed to realize smart agriculture.

[0014] Therefore, one objective of various embodiments of the present invention is to design a two-layer hybrid agricultural network architecture using agricultural sensors, UAVs, and cloud servers, wherein the agricultural sensors and UAVs form a first-layer network, and the UAVs and cloud servers form a second-layer network, wherein the UAVs act as conventional communication infrastructure and support two communication protocols: short-range communication protocols such as IEEE 802.11 or IEEE 802.15.4 used by the agricultural sensors and long-range communication protocols such as LTE / 5G used by the cloud servers. In the first-layer communication network, the agricultural sensors use the short-range communication protocol to communicate with the UAVs to transmit sensor data from the agricultural sensors to the UAVs, and in the second-layer communication network, the UAVs and cloud servers use the long-range communication protocol to communicate to route received data from the UAVs to the cloud servers.

[0015] Some embodiments of the present invention are based on the understanding that agricultural sensors are sparsely deployed in agricultural areas. Depending on the application, some types of sensors can be deployed underground, some on the ground, and some above the ground. Therefore, there are different ways to cluster sensors, including location-based clustering, sensor-type-based clustering, and communication connectivity-based clustering.

[0016] Therefore, some embodiments of the present invention cluster agricultural sensors based on geometric location and communication connectivity, so that sensors in the same cluster can form a connected communication network, wherein one or more UAVs can be assigned to collect sensor data from the cluster at given time intervals.

[0017] Some embodiments of the present invention are based on the understanding that farms can deploy different types of sensors to collect different data. Therefore, when to collect data, what data to collect, and how much data to collect may vary. Different sensors can collect data in different ways; some sensors can collect data randomly, while others can collect data periodically. Thus, some agricultural sensors can collect more data, and some agricultural sensors can collect less data.

[0018] Therefore, some embodiments of the present invention treat agricultural sensor data collection as a stochastic process, such that each agricultural sensor collects data accordingly with a probability distribution.

[0019] Some embodiments of the present invention are based on the understanding that agricultural sensors have small memories to store the collected data. Therefore, UAVs must pick up agricultural sensor data in a timely manner to avoid data loss due to memory overflow.

[0020] Therefore, some embodiments of the present invention model data picking as a first Markov decision process (MDP) problem, wherein the state is the size of the agricultural sensor’s buffer (the number of data packets stored), the action is data packet picking, and the cost (reward) is the data packet picking delay.

[0021] Some embodiments of the present invention are based on the understanding that UAVs have both communication capabilities and storage limitations, which means that they can only pick up a limited number of data packets from an agricultural sensor cluster at a given time interval.

[0022] Therefore, it is essential to efficiently pick up data packets from agricultural sensor clusters to avoid data packet loss due to memory overflow. Consequently, some embodiments of the present invention formulate a first Multiplication Table (MDP) problem to find the optimal picking strategy that minimizes average latency and thus avoids memory overflow. Existing relative value iterative methods can be applied to solve the first MDP problem.

[0023] Some embodiments of the present invention are based on the understanding that UAVs have buffering limitations and energy supply constraints. Therefore, efficient data delivery and energy conservation are crucial. Thus, it is impractical for a UAV to travel to a remote cloud server to offload the collected sensor data. Therefore, a collaborative data delivery method is needed.

[0024] Some embodiments of the present invention are based on the understanding that long-distance communication protocols can support peer-to-peer (P2P) communication without the assistance of infrastructure. For example, the LTE / 5G protocol is enabling device-to-device (D2D) communication, which allows LTE / 5G devices to communicate directly without a base station.

[0025] Therefore, some embodiments of the present invention form the second-layer network as a dynamic multi-hop mesh network for coordinated data delivery, consisting of UAVs and cloud servers. The UAVs and cloud servers dynamically form the mesh network using a long-range communication protocol, while the UAVs use a short-range communication protocol to collect sensor data from agricultural sensors in the first-layer network. Using the formed mesh network, the UAVs deliver the collected sensor data to the cloud server within the dynamically formed multi-hop second-layer network.

[0026] Some embodiments of the present invention are based on the understanding that routing data from UAVs to cloud servers in a mesh topology requires a routing strategy. Therefore, a second Markov decision process (MDP) problem is formulated based on the states of UAVs in a second-layer UAV mesh network, where the UAV state includes the number of packets in the UAV buffer and the observed hops, and the UAV action is to select a neighbor and send data packets to the selected neighbor, where the UAV's action space is the set of neighbors.

[0027] Therefore, some embodiments of the present invention formulate a second MDP problem to minimize data delivery latency without buffer overflow. Instead of using a fixed routing protocol, some embodiments of the present invention use multi-agent reinforcement learning (MARL) to solve the second MDP problem, wherein each UAV acts as a learning agent.

[0028] To address this, a Focused Coordinating Multi-Agent Deep Deterministic Policy Gradient (FC-MADDPG) algorithm is provided to implement MARL for solving the second MDP problem. The FC-MADDPG algorithm is a semi-cooperative method, allowing learning agents to share information only with their neighbors. Therefore, each agent uses its own information and neighbor information to train a machine learning model and make decentralized action decisions.

[0029] According to an embodiment of the present invention, a drone (UAV) agent is provided to deliver data packets from agricultural sensors deployed in farmland to at least one cloud server via a two-layer hybrid network architecture consisting of at least one first-layer network and a second-layer network, wherein the at least one first-layer network is formed by an agricultural sensor cluster of agricultural sensors and at least one UAV agent via a first link, and the second-layer network is formed by the UAV agent and at least one cloud server via a second link. UAV agents may include: a first transceiver configured to receive data packets from agricultural sensors via a first link; a second transceiver configured to receive data packets from neighboring UAV agents and route the received data packets to at least one cloud server via a second link; a memory configured to store the first link corresponding to the agricultural sensor and its geographic location, the second link corresponding to a neighboring UAV agent or at least one cloud server, the number of agricultural sensors in the corresponding agricultural sensor cluster, and a computer-executable program including a Focused Coordinated Multi-Agent Deep Deterministic Policy Gradient (FC-MADDPG) algorithm; and a processor configured to execute the steps of the computer-executable program, wherein the steps include: using the first transceiver to collect data via the first link of at least one first-layer network. During each time interval of the process, data packets are received from selected agricultural sensors storing data packets, wherein the data collection process is performed by solving a first Markov Decision Process (MDP) problem formulated based on the state of the UAV agent and the state of the agricultural sensor; a route to at least one cloud server is learned using a second transceiver via a second link of a second layer network by solving a second MDP problem formulated based on the state of the UAV agent, wherein the second MDP problem is solved by using multi-agent reinforcement learning (MARL), wherein the FC-MADDPG algorithm is provided to implement MARL, wherein the execution output of the FC-MADDPG algorithm is a route for each UAV agent; and the learned route is used to route the received data packets from the UAV agent to at least one cloud server using the second transceiver via the second link of the second layer network.

[0030] Furthermore, based on some embodiments of the present invention, a two-layer hybrid network architecture is provided. In this case, the two-layer hybrid network architecture may include: multiple agricultural sensors configured to collect measurement results, store, and transmit data packets of the measurement results, wherein the agricultural sensors are deployed in farmland; multiple unmanned aerial vehicle (UAV) agents, wherein each UAV agent is configured to communicate with one or more agricultural sensors to form at least one first-layer network with the agricultural sensors, the at least one first-layer network being used to receive, store, and transmit data packets collected by the agricultural sensors; and at least one cloud server communicating with at least one UAV agent to form a second-layer network, wherein the second... A layer network is configured to route data packets from UAV agents to at least one cloud server; wherein at least one first-layer network is formed by an agricultural sensor cluster of agricultural sensors and at least one UAV agent via a first link; wherein a second-layer network is formed by UAV agents and at least one cloud server via a second link; wherein each UAV agent includes: a first transceiver configured to receive data packets from an agricultural sensor via the first link; and a second transceiver configured to receive data packets from a neighboring UAV agent and route the received data packets to at least one cloud server via the second link. A memory configured to store a first link corresponding to an agricultural sensor and its geographic location, a second link corresponding to a neighboring UAV agent or at least one cloud server, the number of agricultural sensors in the corresponding agricultural sensor cluster, and a computer-executable program including a Focused Coordinated Multi-Agent Deep Deterministic Policy Gradient (FC-MADDPG) algorithm; and a processor configured to execute the steps of the computer-executable program, wherein the steps include: receiving data packets from selected agricultural sensors storing data packets at each time interval during a data collection process via the first link of at least one first-layer network using a first transceiver, wherein the data packets are obtained by solving a solution based on the agricultural sensor. The data collection process is performed by solving a first Markov Decision Process (MDP) problem based on the state of the UAV agents; a route to at least one cloud server is learned using a second transceiver via a second link of a second-layer network by solving a second MDP problem based on the state of the UAV agents, wherein the second MDP problem is solved by using multi-agent reinforcement learning (MARL), wherein the FC-MADDPG algorithm is provided to implement MARL, wherein the execution output of the FC-MADDPG algorithm is for each UAV agent; and based on the learned route, the received data packets are routed to at least one cloud server using the second transceiver via a second link of a second-layer network.

[0031] The embodiments disclosed herein will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, but generally focus on illustrating the principles of the embodiments disclosed herein. Attached Figure Description

[0032] [ Figure 1 ]

[0033] Figure 1 This is a schematic diagram illustrating a two-layer hybrid agricultural network architecture with three agricultural sensor clusters, three UAVs and a cloud server according to some embodiments of the present invention, wherein the three first-layer networks are respectively formed by the three agricultural sensor clusters and three assigned UAVs, and the second-layer network is formed by the three UAVs and the cloud server. [ Figure 2A ] Figure 2A An example of a first-layer network having one UAV and N agricultural sensors according to some embodiments of the present invention is shown; [ Figure 2B ] Figure 2B An example of sensor data picked up by a UAV in an agricultural sensor cluster at time T1 is shown according to some embodiments of the present invention; [ Figure 2C ] Figure 2C An example of sensor data picked up by the same UAV in the same agricultural sensor cluster at time T2 is shown according to some embodiments of the present invention; [ Figure 3 ] Figure 3 The time interval division for agricultural sensor data transmission is described according to some embodiments of the present invention; [ Figure 4A ] Figure 4A An example of a Layer 2 network topology with five UAVs and one cloud server at time T1 according to some embodiments of the present invention is illustrated. [ Figure 4B ] Figure 4B An example of a Layer 2 network topology with the same five UAVs and cloud servers at time T2 according to some embodiments of the present invention is illustrated; [ Figure 5 ] Figure 5The provided Focused Coordinating Multi-Agent Deep Deterministic Policy Gradient (FC-MADDPG) algorithm according to some embodiments of the present invention is shown, wherein the FC-MADDPG algorithm is an implementation of Multi-Agent Reinforcement Learning (MARL). [ Figure 6 ] Figure 6 A flowchart of the FC-MADDPG algorithm according to some embodiments of the present invention is shown; [ Figure 7A ] Figure 7A The multi-agent learning routing topology for a second-layer network at time T1 is shown according to some embodiments of the present invention. [ Figure 7B ] Figure 7B This illustrates a multi-agent learning routing topology for the same Layer 2 network at time T2 according to some embodiments of the present invention; [ Figure 8 ] Figure 8 An example of a 30-UAV mesh deployment used in performance evaluation according to some embodiments of the present invention is shown; and [ Figure 9 ] Figure 9 Description of some embodiments of the present invention Figure 8 The data delivery latency results of different machine learning algorithms for the UAV mesh network are shown. Detailed Implementation

[0034] While the accompanying drawings illustrate the embodiments currently disclosed, other embodiments are contemplated, as noted in the discussion. This disclosure presents exemplary embodiments by way of representation, not limitation. Those skilled in the art can devise many other modifications and embodiments falling within the scope and concept of the principles of the embodiments currently disclosed.

[0035] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, apparatus and methods are shown in block diagram form only to avoid obscuring this disclosure.

[0036] As used in this specification and claims, the terms “e.g.,” “such as,” and “like,” as well as the verbs “comprising,” “having,” “including,” and other verb forms thereof, when used in conjunction with a list of one or more components or other items, are each interpreted as open-ended, meaning that the list is not considered to exclude other additional components or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0037] In an era of explosive global population growth, sustainable and efficient agricultural practices have become essential. The Internet of Things (IoT) paradigm has already played a significant role in many urban applications such as smart meters and smart cities. Many use cases in the agro-industry are well-suited for IoT, including soil and crop monitoring, greenhouse automation, and predictive analytics for smart farms. IoT can bring many benefits to agriculture, such as better decision-making using data, risk reduction, cost management, and improved business efficiency. However, realizing smart agriculture also faces many challenges. (1) Lack of communication infrastructure is a major problem for smart farms. Due to cost considerations, agricultural sensors are often equipped with short-range communication radios operating in unlicensed frequency bands, which means that these sensors may not be able to communicate directly with remote cloud servers. Therefore, communication relays are needed. (2) Poor communication connectivity is another problem. Farmers rely on sparsely distributed sensor networks to collect data. Due to the limitations of sensor resources, the collected data may need to be delivered to remote data centers for processing and utilization. For this purpose, communication links should be reliable enough to withstand harsh weather conditions and ensure non-destructive operation. However, most farms are located in remote rural areas where communication connectivity may not be strong enough to facilitate rapid data transmission. In addition, communication links may be blocked by crops and other physical barriers. Efficient scheduling of sensor data transmission presents another challenge. (3) Sensors do not have large memory to store large amounts of data. Therefore, data needs to be transmitted in a timely manner. In other words, how to schedule data transmission to avoid data loss due to buffer overflow and reduce data delivery delays becomes crucial. There are existing technologies for developing smart agriculture sensor technologies. SoilTech is a joint research project of several US universities to develop sensor systems capable of measuring dynamic variables in managed and unmanaged soils in situ and remotely. There are also existing technologies for developing platform technologies. To overcome the challenge of poor communication connectivity, Microsoft FarmBeats aims to use idle TV frequencies to transmit information.

[0038] However, due to the lack of conventional communication infrastructure, networking technologies that act as a bridge between sensors and platforms have been less studied. Agricultural network infrastructure is not yet widely available. Therefore, this factor must be considered in the development of networking technologies. Connecting agricultural sensors with wires is impractical. Therefore, sensors communicate using wireless communication interfaces. Different types of wireless communication technologies exist; for example, LTE / 5G and LoRa are long-range wireless communication technologies. However, using LTE / 5G technology in licensed frequency bands for agricultural sensors is expensive. LoRa can operate in unlicensed frequency bands, but it is not a standardized technology and has a very low data transmission rate. In addition, unregulated channel access can lead to severe interference. Therefore, practical communication technologies are short-range communication protocols that are expected to operate in unlicensed frequency bands, such as IEEE 802.11 (Wi-Fi) and IEEE 802.15.4 (Wi-SUN). The maximum communication distance of these short-range communication protocols is 1 km, which is insufficient for rural agricultural sensors to send data to remote data processing centers. Therefore, a delivery network is needed. Due to the lack of conventional communication infrastructure, an innovative approach is required.

[0039] This invention applies the promising capabilities of the IoT paradigm to revolutionary smart agriculture, a key aspect of contemporary agricultural technology. To address the lack of conventional communication infrastructure and inefficient data transmission due to unstable wireless connections, a novel smart agriculture architecture utilizing unmanned aerial vehicles (UAVs) is provided. This UAV-assisted agricultural network architecture significantly enhances the reliability and efficiency of data transmission from agricultural sensors to cloud servers. In this architecture, UAVs act as conventional communication infrastructure by forming a dynamic router network when agricultural sensor data collection is needed. In other words, to reduce costs, UAVs operate only when required. However, this innovative approach presents its own set of challenges, particularly in scheduling data transmission from agricultural sensors to UAVs and in implementing multi-hop routing from UAVs to remote cloud servers. Therefore, the following techniques are provided to address these challenges: 1. An innovative UAV-assisted dynamic two-layer hybrid smart agriculture network architecture is provided, wherein UAVs are used to address the lack of feasible communication infrastructure in rural farmland.

[0040] a. The UAV and agricultural sensor clusters dynamically form a first-layer multi-point-to-point (MP2P) network to transmit data from the agricultural sensors to the UAV.

[0041] b. The UAV and cloud server dynamically form a second-layer mesh network to route data from the UAV to the cloud server.

[0042] 2. The problem of scheduling data transmission from agricultural sensors to UAVs in a first-layer network is formulated as a first Markov decision process (MDP) problem.

[0043] 3. The problem of routing the collected sensor data from the UAV to the remote cloud server in the second-layer network is defined as the second MDP problem.

[0044] 4. Apply Multi-Agent Reinforcement Learning (MARL) to solve the second MDP problem in the second layer network.

[0045] 5. Therefore, the Focused Coordination MADDPG (FC-MADDPG) algorithm is introduced to implement MARL. This novel algorithm is designed to reduce communication overhead and mitigate the vulnerabilities associated with centralized control algorithms, thus providing a decentralized and stable solution to the challenges of UAV-assisted smart agriculture networks. It defines an innovative reward function, integrates an attention model to allow the UAV agent to autonomously determine the relevance of information from neighboring agents, and applies the Gumbel-Softmax method to approximate the gradient of the discrete stochastic policy.

[0046] Strengthen learning and initial

[0047] Reinforcement learning (RL) describes the process by which an agent acquires knowledge to make decisions through direct interaction with its environment. In RL, the agent first perceives the current state of the environment. Based on this observation, it decides what action to take. After the action, the environment responds, which in turn causes the agent to receive feedback in the form of a reward. The fundamental goal of RL is for the agent to develop policies that effectively maximize the total expected reward over time.

[0048] Elements of a Markov Decision Process (MDP)

[0049] The fundamental model for RL problems is the Markov Decision Process (MDP), which provides a mathematical framework for modeling decisions in situations where outcomes are uncertain and may be influenced by the actions of the decision-maker. MDPs can be modeled as tuples. Decision-making in systems with a finite set of states is particularly useful. , where state Indicates the agent's time t The state of time, for example, the number of stored data groups; a finite set of actions. Among them, actions Indicates the agent's time t Actions during the process, such as sending data packets or not sending data packets; state transition kernel Define the probability of moving from one state to another given an action, i.e., This indicates that the agent is taking action. Next from state s Transition to state s’ The probability, for example, if the current state has 10 packets, then after sending one packet, the next state has 9 packets; reward function ,in, This indicates that the agent is in a state. s Take action below The expected reward (cost) in the case of a successful packet transmission, for example, the receiving... +1 bonus for unsuccessful packet transmissions. -1 reward; and discount factor This is typically multiplied by future rewards to mitigate the impact of future rewards. For any policy... State-action value function (i.e., in strategy) The cumulative reward for a given state-action pair can be defined as follows: (1) in, It is a strategy The expectations below, and Based on time step t The reward received. In RL, the agent's goal is to find a strategy that maximizes the expected cumulative reward. This is equivalent to finding the optimal state-action value function; (2) The optimal strategy is a greedy strategy, in which the agent chooses the action with the highest state-action value. Q The action.

[0050] Reinforcement learning algorithms

[0051] (1) Value-based approach: Value-based approach focuses on estimating each state in the environment. or state-action pair The most common approach is to learn the value function. or The value estimate is the expected reward (cumulative reward) derived from a given state or state-action pair. Key algorithms in this class include Q-learning and Deep Q-Networks (DQNs). These methods typically use a technique called temporal difference (TD) learning to update the value estimate. The main objective here is to maximize the value function, which indirectly learns the optimal policy that satisfies the Bellman equation. ,as follows: (3) (2) Policy gradient-based methods: In contrast to value-based methods, policy gradient methods aim to directly optimize the policy. ,in, These represent the parameters of the strategy. The goal is to maximize the expected return. Maximize, where, It is a cumulative reward. These methods adjust the policy by calculating the gradient indicating how to change policy parameters to increase the expected return, where, regarding gradient It is given by the following formula: (4) This gradient is used to update the policy parameters in the direction of higher expected reward. A well-known example is the REINFORCE algorithm, which uses the Monte Carlo method to estimate the policy gradient. Other advanced techniques include Trust Region Policy Optimization (TRPO) and Proximity Policy Optimization (PPO) to improve the stability and efficiency of training. These methods are particularly useful in high-dimensional or continuous action spaces.

[0052] (3) Actor-evaluator based approach: The actor-evaluator approach combines elements of value-based and policy gradient approaches. The "actor" is the policy network that determines the optimal action to take, represented by the policy function. The "evaluator" is a value network that assesses the actions taken by the actor, represented as... or ,in, These are the evaluator's parameters. The evaluator estimates the value function, which is used to update the policy (actor) more efficiently. The actor updates its policy parameters based on the following gradients. : (5) in, It is the dominance function, often calculated as The rater's parameters are typically updated based on TD error or other value-based methods. Examples include Deep Deterministic Policy Gradient (DDPG) and Asynchronous Advantage Actor-Evaluator (A3C). These methods stabilize the training process by reducing the variance of the policy gradient using the evaluator's value estimate. They are effective in a variety of complex environments, balancing the strengths of both value-based and policy-based approaches.

[0053] Multi-agent reinforcement learning (MARL)

[0054] RL can also be performed by multiple agents. Multi-agent reinforcement learning (MARL) is a branch of RL that deals with environments where multiple agents interact with each other and simultaneously with the environment. Unlike single-agent RL, which focuses on learning the optimal policy for a single agent, MARL involves learning the optimal policy in the context of the policies and actions of other agents within a partially observed environment. This makes the problem significantly more complex due to the dynamically changing environment caused by agent interactions. Multi-agent DDPG is an extension of the DDPG algorithm to a multi-agent setting. In the multi-agent version, each agent... k It has its own network of actors and the network of evaluators The agent decides the best action to take given the current state, while the evaluator assesses the action by estimating the Q-value. However, due to the varying degrees of complexity in each agent's actions... k It can only have the global state of the environment. s Local observation results ,in, Therefore, depending on the specific implementation and problem setting, agents can share information or learn collaboratively. A common approach in Multi-Agent Programming (DDPG) is to utilize centralized training with decentralized execution. This means that during training, the evaluator has access to the actions and states of all agents, but during execution, each agent's actor network learns based on its local observations. Operating independently. In the multi-agent DDPG framework, each agent's evaluator estimates the value of a state-action pair, considering not only its own actions but also the actions of other agents. (For agents...) k The policy parameters are represented as And the policy for all agents is represented as Actions for all agents are represented as The gradient of the stochastic policy for each agent can be expressed as: (6) here, This represents the centralized action-value function estimated by the evaluator for each agent, captured in the state. s Take action below The expected return. The goal is to achieve this by following the strategy parameters. Optimize the gradient of expected return To maximize expected returns. For DDPG, each agent... k Actors optimize their own strategies And parameters regarding the evaluator The certainty gradient of the expected return is given by the following equation: (7) in, It serves as a replay buffer for storing past experiences. Furthermore, to calculate the evaluator's estimate of the value function, a target value function is used, which is the expected reward for taking an action in the current state and subsequently following a target policy: , (8) in, It is the set of target policies for all agents, where, It is used to calculate the target value for stable training.

[0055] Attention mechanism

[0056] The attention mechanism in neural networks can be mathematically described using several key components: query, key, value, and attention function. These components work together to produce a weighted sum of values, where the weight assigned to each value depends on the compatibility of the query with the corresponding key. m In the case of a sequence of n elements, the steps of the attention mechanism can be described as follows: Vector transformation: for each element in the sequence i ,in Input vector Transformed into a query ,key Sum : (9) Attention weight calculation: using the softmax function to... With all keys in the sequence Compare to calculate attention weights for each element This yields the distribution of weights with a sum of 1: (10) Here, softmax is applied to the vector of original attention scores to obtain normalized attention weights. .

[0057] Context vector: The context vector for each element. Calculated as a vector of all values The weighted sum, where the weights are determined by... Given: (11) in, It corresponds to the first in the sequence.j Element Weights. Context vector. It is the key output of the attention mechanism because it encapsulates the information from the element. i The most relevant information in the input sequence. This process allows the model to selectively focus on different parts of the input when processing each element, similar to how human attention focuses on different parts of a scene or dialogue.

[0058] Gumbel-Softmax distribution

[0059] The Gumbel-Softmax method is used to approximate gradients of discrete distributions. It assumes the policy needs to take steps according to... From strategy distribution The sampling process. Here, each... And satisfy Policy selection needs to be approximated by the indistinguishability of the discrete properties of the policy distribution. In order to approximate this process using class probabilities... Sampling from a categorical distribution can be done using the Gumbel-Softmax distribution. To sample from a Gumbel(0,1) distribution, the noise from the Gumbel(0,1) distribution is first calculated as follows: , .

[0060] Then, the Gumbel-Softmax sample is given by the following formula: .

[0061] in, It is a temperature parameter that controls the approximate smoothness. With Samples from the Gumbel-Softmax distribution become closer to the one-hot vector. Gumbel-Softmax sampling is provided, and then an approximate discrete action is obtained.

[0062] System Model

[0063] Some embodiments of the present invention provide a two-layer hybrid smart agriculture network architecture, wherein the first layer network is dynamically formed by agricultural sensor clusters and assigned UAVs via short-range communication protocols, and the second layer network is dynamically formed by all UAVs and cloud servers via long-range communication protocols. Figure 1An example of a two-layer hybrid smart agriculture network architecture 100 is shown, which includes a cloud server 101, three UAVs 102, 103, and 104, and three agricultural sensor clusters 105, 106, and 107. Each UAV and its corresponding sensor cluster form a first-layer network 110 using a short-range communication link 108. Thus, three first-layer networks are formed. The three UAVs and the cloud server form a second-layer network 111 using a long-range communication link 109.

[0064] The second-layer network 111 is a UAV-assisted relay network in a rural agricultural environment. Because agricultural sensors are typically equipped with short-range communication radios, deployed agricultural sensors may not have a direct communication link with a remote cloud server. Furthermore, conventional communication infrastructure may not be available in rural areas. Therefore, this invention uses UAVs instead of conventional communication infrastructure (such as base stations in cellular communication networks) as node proxies to relay data from agricultural sensors to a remote cloud server for sensor data processing. The UAVs consist of a set... Index. Based on the geographical location and communication links of the deployed agricultural sensors, the overall sensor system is divided into... K Each cluster consists of [number] clusters. Without loss of generality, each cluster consists of [number] clusters. N It consists of several sensors. During the sensor data relay process, the first... k A UAV hovers over the cluster in a specific geographical area. k From the cluster k Sensors collect data and relay it to a cloud server. However, some geographical areas may be far from the cloud server, so associated UAVs in these areas may not be able to communicate with the cloud server or may not have a reliable communication link with it. This requires UAVs to cooperate in relaying data to the cloud server. Therefore, UAVs dynamically form a mesh network to relay sensor data. Then, the goal of each UAV is to find the optimal multi-hop route to the cloud server.

[0065] There are two operations involved in relaying data from agricultural sensors to a remote cloud server: the first operation is transmitting data from the sensor to the UAV, and the second operation is relaying data from the UAV to the cloud server. The first operation is performed in the first-layer network, and the second operation is performed in the second-layer network. However, both operations can be performed simultaneously, meaning the UAV can relay the collected data to the cloud server while simultaneously collecting data from the sensor.

[0066] For the first operation, UAV 200 was assigned to serve the service provided by N A sensor cluster 201 consisting of agricultural sensors, such as Figure 2A As shown, each sensor n According to probability Agricultural data, such as soil temperature, is measured probabilistically and independently. The UAV 200 collects data from the sensor cluster 201 using a short-range communication link 108. Although it has limited capacity... C The UAV can only relay combined data from a subset of sensors, but the capacity is a combination of factors that take into account both communication capabilities and storage limitations. Figure 2B The UAV 200 is shown at time T1 from sensor 1, sensor 2 and sensor 3. N -1 collects data, however, as Figure 2C As shown, UAV 200 detects signals from the sensor at time T2. n and sensors N Data is collected. Therefore, time can be divided into time intervals of 300, such as... Figure 3 As shown. The challenge is how to extract data from each time interval. N Sensor selection C The goal is to minimize the average latency of agricultural sensor data relay using multiple sensors. This optimization problem involves the need for efficient resource allocation, consideration of probabilistic data generation, and prioritization of real-time data delivery to achieve the critical objective of minimizing the average latency of agricultural sensor data relay.

[0067] For the second operation in the Layer 2 network, the UAV traverses its corresponding cluster area to collect data over T time intervals, resulting in dynamic topology changes in the mesh network formed by the UAVs and cloud servers. To effectively manage data transmission in this dynamic environment, the development of dynamic routing technology is necessary. The routing process directly impacts the capacity of the UAVs used for data packet scheduling during each time interval. C Traditionally, route discovery is performed using protocol-based methods. Recently, machine learning-based routing techniques have been proposed and may outperform traditional methods in complex network environments. RL and deep learning (DL) methods can be used to design and adapt routing techniques that efficiently allocate UAV resources and minimize latency, taking into account evolving network topologies and data generation probabilities. UAVs and cloud servers form a dynamic Layer 2 network. Figure 4A An illustrative example of a Layer 2 network topology consisting of five UAVs and a cloud server 101 at time T1 is shown, wherein each UAV is associated with a data buffer 400, and a long-distance communication link 109 is used for communication. However, due to the dynamic nature of UAVs, as... Figure 4B The second-layer network topology shown at time T2 for the same set of UAVs and the cloud server is different. This dynamic mesh network framework offers the potential to optimize data transmission from the VAVs to the cloud server 101, improve resource utilization, and ensure timely and efficient data delivery in the ever-changing agricultural sensor network.

[0068] The problem of data packet transmission from the first-layer sensor to the UAV network is formulated.

[0069] Consider as Figure 2A The first layer of the agricultural network shown is represented as a set, where the agricultural sensor cluster 201 is represented as a set. of N It consists of several agricultural sensors, among which... Agricultural sensors measure different data and generate various packets from the environment that need to be sent to a remote cloud server for processing. For ease of explanation, it can be assumed that all data packets generated by the sensors have a unit size. A UAV 200 with a limited capacity can serve a maximum of [number missing] data packets within a time interval. C The UAV 200 has 1,000 sensors, where capacity takes into account communication capabilities and storage limitations. The goal of the UAV 200 is to determine which sensors to serve at each time interval to minimize the cumulative average packet delivery latency experienced by the agricultural sensors.

[0070] Group generation and delivery model: Time is divided into multiple units, each unit is called an "interval", which is represented as .sensor In each time interval t With probability Generate groups. (By sensor) n The generated data packets are buffered in a queue that stores data generated by the sensors. n Multiple incomplete data groups were generated until time. t ,like Figure 2A As shown. (With time) t The queue length associated with the number of such data groups is expressed as: The rationale for this model is that the number of data packets generated by the sensor cluster 201 can exceed the service capacity of the wireless UAV. Therefore, data packets generated by each agricultural sensor may not be served immediately, resulting in a delay associated with the agricultural sensors receiving the generated data packets delivered to the UAV 200. Another consideration is that the wireless channel between the UAV 200 and the agricultural sensors is unreliable. A queuing model considering the delay experienced by the agricultural sensors is a motivation. Due to the unreliability of the wireless channel, the delay from the agricultural sensors... n Data packets delivered to the UAV 200 are random events, and successful transmission is probabilistic. occur.

[0071] Therefore, the data packet delivery problem in the above model is formulated as an MDP process (the first MDP problem).

[0072] Status: Agricultural sensor cluster in time t The state is represented as ,in, In agricultural sensors The number of incomplete data packets in the buffer. To ensure the stability of the Markov chain, assume... ,in, This is the queue capacity, i.e., the maximum number of packets that can be buffered by agricultural sensors. The associated state space is represented as .

[0073] Action: at each time t For each agricultural sensor n UAVs must make decisions about whether or not to serve them. Used to indicate time t For agricultural sensors n The actions taken, among which, Indicator Sensor n Being served, and Indicator Sensor n Not served. It is a set of decisions that can be used for each action, and let It is aimed at N The decisions of each agricultural sensor are composed of a vector. The capacity constraints of the communication link and UAV storage mean that... The following constraints must be met; (12) The goal is to design the state of the sensor cluster. Mapping to decision strategy ,Right now, .

[0074] Transfer nucleus: the n The status of each agricultural sensor queue can be determined based on the actions taken by the UAV and the agricultural sensors. n The probability of group generation from time t arrive t +1 from Change to or Or remain unchanged. The detailed transfer is as follows: (13) It is easy to verify that the sum of the probabilities for the three scenarios equals 1.

[0075] Data packet delivery problem: According to Little's law, minimizing the average latency faced by agricultural sensors is equivalent to minimizing the average number of accumulated uncompleted data packets in the sensor cluster. Let... It is an agricultural sensor n In time t The instantaneous cost generated, so that in time t The cumulative cost generated in a sensor cluster is given by the following formula: (14) By choosing this instantaneous cost, the average cost generated in the system is proportional to the average time delay faced by the agricultural sensors. The goal is to derive a strategy for solving the following MDP: Contracted to (15) The subscript indicates that the strategy is... The resulting measurement reflects the expected fact, and It is the set of all feasible strategies. Problem (15) is then referred to as the “original MDP”. Since it is an infinite time-domain average cost problem, it can be solved using existing relative value iteration methods.

[0076] Data packet routing problem in a second-layer UAV mesh network

[0077] Figure 4A An example of a second-layer UAV mesh network is shown, where long-distance communication links 109 are used for communication. The UAV mesh network can be constructed as an undirected graph. ,in, It is a set of nodes, and It is a set of links. Nodes can be UAV nodes or cloud server nodes. Only when a node... i and nodes j When nodes can communicate directly with each other, i With nodes j Links only exist between them. 。 is represented as node i The network consists of a set of neighboring nodes. These links are bidirectional in data routing, while the connection between the UAV and the cloud server is unidirectional, meaning there is no sensor data flow from the cloud server to the UAV. This distinction in link directionality is crucial for routing and data packet transmission paths. In this network, data packets are delivered to the cloud server via a series of relay routers (UAVs). This process is called multi-hop routing. Specifically, each UAV maintains a queue to store data packets collected from agricultural sensors in its associated cluster and data packets received from neighboring UAVs. UAVs manage their data packets based on a first-in, first-out (FIFO) principle. This means that packets are processed in the order they are received, ensuring fair and orderly processing of data without giving some packets priority over others.

[0078] Each UAVk The goal is to find a routing strategy. To minimize the average delay, i.e., to maximize the expected cumulative reward in the RL aspect. Similarly, this routing problem can be formulated as another MDP process (the second MDP problem).

[0079] Status: The UAV mesh network will be in time... t The state is represented as ,in, It is for UAV The state represents the queue length, i.e., the number of data packets stored, and the observed hops (HO). Therefore, if the cloud server is its neighbor, the UAV sets HO=1. Otherwise, the UAV sets HO=1 + minimum HO. HO provides the UAV with a reference for next-hop router selection. Since each UAV has limited storage, let... It is the maximum queue length.

[0080] Observation Space: Sharing state across the mesh network is impractical due to communication overhead and computational complexity. Nor is it necessary to share state across networks, as the state of remote UAVs may not contribute to decision-making. Therefore, UAVs share their state with neighbors only via broadcast transmission to reduce communication overhead. Thus, in a UAV mesh network, each UAV can only observe its own state and the state of its neighbors, representing only a partial observation of the entire environment. k The observation results are expressed as follows: , in, It is a UAV intelligent agent k state, It is a UAV intelligent agent k The set of neighbors, if the neighbors j If it is another UAV agent, then It is the neighbor's UAV intelligent agent The state, and if the neighbor j If it is a cloud server, then It is an empty set. Therefore, the global state of the environment is a joint observation result, that is, .

[0081] Action: at each time t UAV k It must decide which neighboring node it should choose. j The decision is made as the next-hop router. (This is represented as...) UAV k In time t The action. Therefore, if adjacent nodes are selected. j Then let Therefore, for UAVs K The size of the action space is equal to the number of adjacent nodes, that is, .make It is aimed at the whole K A vector composed of the decisions of each UAV.

[0082] Reward Function: Reinforcement algorithms (RL) can transform difficult optimization problems into problems of maximizing expected cumulative rewards through appropriate reward function design. When designing a reward function for the routing problem in UAV mesh networks, several key aspects need to be considered: (1) Queue length: UAV k queue at the location The length reflects the number of packets waiting to be sent from that node. A lower value indicates lower latency, so queue length is penalized when designing the reward function.

[0083] (2) End-to-end latency (UAV to cloud server latency): This is achieved through the latency from the node i To the node j Transmission time To quantify. End-to-end latency is a key metric in network performance, reflecting the time it takes for packets to travel through the network from source to destination. Lower values ​​of these values ​​typically contribute positively to the reward function because they indicate faster transmission, which is generally desirable.

[0084] (3) Packet delivery success / failure: This is determined by the acknowledgment (ACK) signal. The indicator can be either 0 or 1. The signal is a binary indicator, where 1 typically indicates successful packet delivery, and 0 indicates failure. Successful delivery. Increase rewards, while failure The reward will be reduced. (For use) The UAV acknowledges the transmitter upon successfully receiving a data packet from its neighbor. Each UAV starts a timer to wait for an ACK packet when it sends a data packet. If the timer expires without receiving an ACK, the transmission is considered a failure. Otherwise, the transmission is considered successful.

[0085] (4) Congestion: The buffer length from any UAV is close to the maximum value. Inferring network congestion from signals This indicates that when the buffer length exceeds the threshold... Congestion can lead to packet delays, increased buffer occupancy, and even packet loss. Therefore, the reward function should be designed to penalize congestion, as it implies a deterioration in network performance.

[0086] (5) Destination reached: This is determined by the signal This indicates that the signal is again a binary value of 0 or 1. Here, the signal... This indicates whether the packet has reached the cloud server. Reaching the cloud server is the primary objective, therefore... This will contribute positively to the reward. (For the purpose of using signals) The mechanism allows cloud servers to distribute and receive combined packets of information across networks.

[0087] Based on the above key factors, a novel reward function is provided as follows: , in, and It is a positive weight scalar. In the routing setup under consideration, each UAV aims to maximize its individual reward at the expense of other UAVs. This is typical in game theory scenarios. For competitive MARL, each UAV... k The goal can be written as: (16) in, In addition to UAV k All other UAV actions, It's a strategy. It is a discount factor, and T This refers to the data delivery duration. Note that UAV... k The optimal strategy depends on global state information And all other actions of UAVs, due to communication overhead, UAV k Knowing this might be impractical. The goal is to provide an approximate MARL solution that requires only partial local observations.

[0088] Focus on Coordinated Multi-Agent DDPG (FC-MADDPG) Algorithm

[0089] Common, conventional techniques for managing dense UAV mesh network routing, as in (16), rely on multi-agent DDPG (MADDPG) algorithms, which typically have significant drawbacks. A major challenge is the substantial communication overhead due to the continuous need for information exchange between UAVs, which depletes network resources, particularly in bandwidth-constrained scenarios. Another challenge is the computational complexity resulting from the sharing of transient states across the network. It's possible that the states of UAVs that are geographically distant may not contribute positively. Furthermore, these traditional approaches often rely on a central controller for coordination, creating a bottleneck in terms of computational load and introducing the risk of single points of failure, especially as the network expands. This centralized approach also struggles with adaptability in dynamic environments, thus limiting the overall efficiency and robustness of the system.

[0090] To overcome these limitations, this invention provides the Focused Coordinating Multi-Agent Projection Generation (FC-MADDPG) algorithm. This algorithm innovatively integrates an attention mechanism, allowing each UAV to autonomously determine the relevance of information from neighboring agents, thereby significantly reducing unnecessary communication overhead. By decentralizing the decision-making process, the FC-MADDPG algorithm eliminates the need for a central controller, improving the scalability and resilience of the UAV network. This attention-guided algorithm is adaptable to changing environments and network topologies, ensuring that UAVs focus on processing the most relevant information. Therefore, the FC-MADDPG algorithm offers a stable, efficient, and scalable solution, circumventing the shortcomings of conventional methods and providing significant progress in the management of dense UAV networks.

[0091] The MADDPG algorithm is an extension of the DDPG algorithm for multi-agent environments. The original DDPG algorithm is a model-free, non-policy actor-evaluator method designed for environments with continuous action spaces. When adapting DDPG to multi-agent systems such as UAV mesh networks, the algorithm needs to handle interactions between agents and partial observability of the environment. Probabilistic methods can not only be applied to the MADDPG framework, but also the gradient of discrete stochastic policies can be approximated using a method called Gumbel-Softmax.

[0092] The following is a detailed overview of the FC-MADDPG algorithm. Figure 5 The FC-MADDPG algorithm is shown, which includes the following steps.

[0093] Initialization: for each agent k By weight Initialize the actor network And by weight Initialize the evaluator network Furthermore, for each intelligent agent... k By weight Initialize the target network And by weight Initialize the target network Since FC-MADDPG is a non-policy method, each agent... k A replay buffer can be stored to hold the experience tuples of each agent and its neighboring agents. It is initialized to an empty set.

[0094] Round-based learning: At the start of each round, the environment is reset to its initial state, and each agent... k Receive initial observation results .

[0095] At each step, each agent k From his neighbor j Receive observation results and actions For each agent, an evaluator network is used. Updated to consider valid states and actions: (17) in, These are the parameters of the evaluator network. Specifically, within the evaluator network, the agent... k Utilize attention mechanisms to compute from each neighboring agent j The weighted sum of observations and actions. (Agent) k The output of the attention layer is represented as: (18) The attention module follows (9)-(11) and These are the parameters of the attention model.

[0096] Similarly, for intelligent agents k Actor network (strategy) Based on local observation results Based on the effective output of the attention module, both are updated as follows: (19) in, These are the parameters of the actor network.

[0097] Then, the intelligent agent k Using actor networks based on valid state information Choose an action And add exploration noise This encourages strategy exploration. After applying the chosen action to the environment, each agent receives new state observations, a reward, and information on whether the turn has ended. It then transfers the tuple. Stored in the playback buffer middle.

[0098] from Random sampling and mini-batch transfer. For each agent. k The evaluator is updated by minimizing the loss between its prediction and the target value calculated using the target network. The agent is updated using the sampled policy gradient derived from the evaluator's output. To be received The target network is updated by mixing the weights of the target network and the main network. and If a terminal state is reached or the maximum number of time steps is exceeded, the environment is reset for the next round. A set of optimization strategies for each UAV ensures efficient routing and stable mesh network performance.

[0099] Figure 6 A flowchart of the FC-MADDPG algorithm is shown. Each UAV agent trains a reinforcement learning model 600 based on a defined optimization problem 601 using its own observations and those of its neighbors. After training, the UAV agent decides 602 which neighbor it will forward its data packets to. Then, the UAV agent sends 603 data packets using the learned route, updating the number of stored data packets, observation hop count, and reward. The UAV agent also shares the number of stored data packets, actions, and observation hop counts with its neighbors; for example, UAV agent 5 broadcasts 604 its information to its neighbors, and UAV agent 3 receives 605 information from its neighbors.

[0100] Once the action policy is learned, the UAV sends its data to the learned next-hop neighbor. Figure 7A An example of the policy learned by the second-layer network at time T1 is shown, wherein UAV 1 and UAV 2 send their data packets directly to cloud server 101, UAV 3 forwards its data packets to UAV 1, UAV 5 sends its data packets to UAV 2, and UAV 4 forwards its data packets 701 to UAV 1 at time T1.

[0101] However, due to the dynamic nature of UAVs, the learned policy changes. For example, Figure 7B The learned policies for the same set of UAVs differ at time T2, where UAV 1 and UAV 2 send their data packets directly to cloud server 101, UAV 3 forwards its data packets to UAV 1, UAV 5 sends its data packets to UAV 2, however, UAV 4 forwards its data packets to UAV 2 at time T2.

[0102] Performance evaluation

[0103] A comprehensive performance evaluation has been conducted. The FC-MADDPG algorithm of the present invention is compared with three benchmark algorithms: (1) the popular Deep Q Network (DQN) algorithm; the centralized MADDPG algorithm, in which agents are selected to perform centralized training by collecting information from all agents, however each agent makes decisions independently; and (3) the independent MADDPG algorithm, in which each agent performs training and makes decisions independently without information sharing.

[0104] Figure 8A UAV mesh network topology with 30 UAVs is shown, where the 30 UAV nodes are randomly deployed in an area of ​​1 km by 1 km, with the cloud server located outside the UAV node deployment area. An LTE / 5G base station is integrated into the cloud server as a node, and UAV 10 and UAV 29 are directly connected to cloud server 101.

[0105] The simulation setup for performance evaluation of the FC-MADDPG algorithm incorporates a detailed model of packet processing and routing within a UAV mesh network, characterized by the following parameters and rules: Each UAV, acting as both a router and a learning agent, is equipped with a buffer that stores incoming data packets for further transmission. This buffer is limited to a maximum capacity of 10 packets to ensure controlled packet flow and prevent buffer overflow under normal operation. Each UAV is responsible for processing a predefined number of packets originating from its designated local coverage area, specifically 25 packets. The UAVs operate synchronously, with each UAV simultaneously sending one packet to its learned next-hop router during each routing decision interval. Packet arrivals at each UAV are modeled as following an exponential distribution, introducing random elements into arrival times to simulate realistic network traffic conditions. A packet loss event is triggered if the time taken to deliver a packet exceeds the duration allocated for each routing decision. Congestion within the UAV's buffer is simulated when the number of stored packets exceeds the buffer's maximum capacity, reflecting real-world network challenges such as bandwidth limitations and high traffic volumes. When the UAV's buffer occupancy drops below a threshold of 5 packets, new packets are proactively acquired, ensuring continuous data flow and optimal utilization of network resources. The simulation round ends once all packets have been successfully relayed to the cloud server (indicating completion of a data transmission cycle within the UAV network).

[0106] Figure 9The data delivery duration, i.e., latency, is shown for different machine learning algorithms. Notably, centralized MADDPG does not produce optimal results. Its use of global information appears excessive in large sparse topologies, leading to inefficiency. In contrast, the FC-MADDPG algorithm exhibits superior performance, outperforming all benchmark algorithms. This improvement is attributed to its ability to focus on relevant information from neighboring nodes, reducing unnecessary processing of redundant data. Furthermore, the DQN algorithm fails to converge, indicating that DQN is unsuitable for complex networks with large heterogeneous action spaces. Specifically, FC-MADDPG requires 712 transport steps to deliver all packets to the cloud server. Centralized MADDPG requires 874 steps, approximately 123% of the FC-MADDPG transport steps. Standalone DDPG requires 917 steps, approximately 129% of the FC-MADDPG transport steps. DQN requires 3684 transport steps, approximately 517% of the FC-MADDPG transport steps.

[0107] The embodiments described above can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, whether provided on a single computer or distributed across multiple computers, the software code can execute on any suitable processor or set of processors. Such a processor can be implemented as an integrated circuit, wherein one or more processors are within an integrated circuit assembly. However, the processor can be implemented using circuitry of any suitable format.

[0108] Furthermore, embodiments of this disclosure can be embodied as a method, examples of which have been provided. Actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments can be constructed in which actions are performed in a different order than those shown, which may include performing some actions simultaneously, even if they are shown as sequential actions in the exemplary embodiments.

[0109] The use of sequential terms such as “first” and “second” in claims to modify claim elements themselves does not imply any priority, precedence, or order of one claim element relative to another claim element or the temporal order of the actions of the method of execution, but is merely used as a label to distinguish one claim element with a specific name from another element with the same name (but for the use of sequential terms) to differentiate claim elements.

[0110] Although this disclosure has been described by way of example of preferred embodiments, it should be understood that various other adjustments and modifications can be made within the concept and scope of the invention.

[0111] Therefore, the purpose of the appended claims is to cover all such variations and modifications that fall within the true concept and scope of the invention.

Claims

1. A drone (UAV) agent, the UAV agent being used to deliver data packets from agricultural sensors deployed in farmland to at least one cloud server via a two-layer hybrid network architecture consisting of at least one first-layer network and a second-layer network, wherein, The at least one first-layer network is formed by the agricultural sensor cluster of the agricultural sensors and at least one UAV agent via a first link, and the second-layer network is formed by the UAV agent and at least one cloud server via a second link. The UAV agent includes: A first transceiver is configured to receive data packets from an agricultural sensor via the first link; A second transceiver is configured to receive data packets from a neighboring UAV agent and route the received data packets to at least one cloud server via the second link. The memory is configured to store the first link corresponding to the agricultural sensor and the geographical location, the second link corresponding to the adjacent UAV agent or at least one cloud server, the number of agricultural sensors in the corresponding agricultural sensor cluster, and a computer executable program including the Focused Coordinating Multi-Agent Deep Deterministic Policy Gradient (FC-MADDPG) algorithm. A processor configured to perform the steps of the computer-executable program, wherein the steps include: The first transceiver receives the data packets from the selected agricultural sensor storing the data packets at each time interval during the data collection process via the first link of the at least one first layer network, wherein the data collection process is performed by solving a first Markov decision process (MDP) problem based on the state of the agricultural sensor. By solving a second MDP problem formulated based on the state of the UAV agents, a route to the at least one cloud server is learned using the second transceiver via the second link of the second layer network, wherein the formulated second MDP problem is solved using multi-agent reinforcement learning (MARL), wherein the FC-MADDPG algorithm is provided to implement the MARL, wherein the execution output of the FC-MADDPG algorithm is for the route of each UAV agent; and Based on the learned route, the second transceiver is used to route the received data packets to the at least one cloud server via the second link of the second layer network.

2. The UAV agent according to claim 1, wherein, The at least one first-layer network uses a short-range communication protocol based on IEEE 802.11 (Wi-Fi) or IEEE 802.15.4 (Wi-SUN) or a combination thereof, wherein the second-layer network applies a long-range communication protocol based on LTE / 5G or LoRa or a combination thereof.

3. The UAV agent according to claim 2, wherein, The short-range communication protocol is used for the first link between the UAV and the agricultural sensor, wherein the long-range communication protocol is used for the second link between the UAV agents and between the UAV agents and the cloud server.

4. The UAV agent according to claim 2, wherein, Based on the connectivity and geometric location of the short-range communication protocol, the agricultural sensors are divided into agricultural sensor clusters in a manner that enables the agricultural sensors within an agricultural sensor cluster to form a fully connected network.

5. The UAV agent according to claim 4, wherein, The number of the at least one first-layer network is equal to the number of the agricultural sensor clusters, wherein the number of agricultural sensors in an agricultural sensor cluster is a variable.

6. The UAV agent according to claim 1, wherein, The at least one first-layer network is a multi-point-to-point (MP2P) network, and the second-layer network is a mesh network.

7. The UAV agent according to claim 1, wherein, The at least one first-layer network is used for collecting data packets from the agricultural sensor cluster by the UAV agent, wherein the second-layer network is used for routing the data packets from the UAV agent to the cloud server, wherein the data collection and data routing are performed simultaneously during the data delivery process.

8. The UAV agent according to claim 1, wherein, In agricultural sensor clusters Depend on N In the case of a sensor assembly, time t The state of the first Markov Decision Process (MDP) problem at time t is: ,in, It is stored in agricultural sensors The number of incomplete data groups at the location.

9. The UAV agent according to claim 1, wherein, In agricultural sensor clusters Depend on N In the case of a sensor assembly, time t The actions of the first Markov Decision Process (MDP) problem at time are ,in, Indicates time t For agricultural sensors The actions taken, among which, Indicator Sensor n Being served, and Indicator Sensor n Not being served.

10. The UAV agent according to claim 1, wherein, In agricultural sensor clusters Depend on N In the case of a system consisting of 1 sensor, the first Markov decision process (MDP) problem is... n The transfer core of the agricultural sensor is represented as ,in, It is a sensor The probability of measuring agricultural data, and It is a sensor The probability of successfully sending data packets to the UAV agent providing the service.

11. The UAV agent according to claim 1, wherein, In agricultural sensor clusters Depend on N In the case of a system consisting of multiple sensors, the first Markov Decision Process (MDP) problem is used to derive the policy. So that Contracted to ,in, C This takes into account the bandwidth of the first link and the storage capacity constraints of the UAV agent, ensuring that it can only serve a limited number of users within a given time interval. N One of the sensors C One sensor.

12. The UAV agent according to claim 1, wherein, exist K In the case of one UAV agent, time t The state of the second MDP problem at that time is , where state This includes the number of data packets stored and the UAV agent. The observed jump (HO).

13. The UAV agent according to claim 1, wherein, exist K In the case of one UAV agent, time t The observation space of the second MDP problem at that time is ,in, It is a UAV intelligent agent The observation results are expressed as ,in, It is a UAV intelligent agent k The state described above, It is a UAV intelligent agent k The neighborhood group, It is the neighbor's UAV intelligent agent The state.

14. The UAV agent according to claim 13, wherein, The UAV intelligent agent k Neighbors j It could be another UAV agent or a cloud server, wherein, if the neighbor j If it is another UAV agent, then The neighbor mentioned j The state, and if the neighbor j If it is a cloud server, then It is an empty set.

15. The UAV agent according to claim 1, wherein, exist K In the case of one UAV agent, time t The action of the second MDP problem at that time is ,in, It is a UAV intelligent agent k In time t The actions at that time, among which, Instructing neighboring agents j Selected as the next-hop router for routing data packets.

16. The UAV agent according to claim 1, wherein, The UAV agents collaboratively solve the second MDP problem by applying multi-agent reinforcement learning (MARL), where each UAV acts as a reinforcement learning agent.

17. The UAV agent according to claim 16, wherein, The reinforcement learning reward design incorporates factors including the number of stored data packets. End-to-end delay Successful packet transmission rate ( ACK Congestion is reflected by the difference between storage capacity and the number of stored data packets. and successful data packet delivery rate So that in, and It is a positive weighted scalar.

18. The UAV agent according to claim 16, wherein, Each UAV agent increases its individual reward at the expense of other UAV agents. Maximize, where UAV agents k The goal was written as ,in, In addition to UAV intelligent agents k All actions of UAV agents other than those mentioned above. It is a discount factor that suppresses the impact of future rewards, and T This refers to the data delivery duration.

19. The UAV agent according to claim 18, wherein, Each UAV agent solves its maximization problem by applying the FC-MADDPG algorithm, wherein the execution output of the FC-MADDPG algorithm is for the route of each UAV agent; wherein the FC-MADDPG algorithm includes initialization and round-robin learning steps.

20. The UAV agent according to claim 19, wherein, During initialization, each UAV agent k By weight Initialize the actor network And by weight Initialize the evaluator network Each UAV agent k By weight Initialize the target network And by weight Initialize the target network .

21. The UAV agent according to claim 18, wherein, In round-based learning, each UAV agent learns from its neighboring UAV agents. j Receive observation results and actions An attention model is applied to recursively update the actor network. and the network of evaluators .

22. A two-layer hybrid network architecture, the two-layer hybrid network architecture comprising: Multiple agricultural sensors are configured to collect measurement results, store and transmit data packets of the measurement results, wherein the agricultural sensors are deployed in farmland; Multiple unmanned aerial vehicle (UAV) agents, wherein each UAV agent is configured to communicate with one or more agricultural sensors to form at least one first-layer network with the agricultural sensors, the at least one first-layer network being used to receive, store and transmit data packets collected by the agricultural sensors; At least one cloud server, which communicates with the at least one UAV agent to form a second-layer network, wherein the second-layer network is configured to route the data packets from the UAV agent to the at least one cloud server; The at least one first-layer network is formed by the agricultural sensor cluster of the agricultural sensors and at least one UAV agent among the UAV agents via a first link; the second-layer network is formed by the UAV agents and the at least one cloud server via a second link; and each UAV agent includes: A first transceiver is configured to receive the data packets from an agricultural sensor via the first link; A second transceiver is configured to receive the data packets from a neighboring UAV agent and route the received data packets to the at least one cloud server via the second link. The memory is configured to store the first link corresponding to the agricultural sensor and the geographical location, the second link corresponding to the adjacent UAV agent or at least one cloud server, the number of agricultural sensors in the corresponding agricultural sensor cluster, and a computer executable program including the Focused Coordinating Multi-Agent Deep Deterministic Policy Gradient (FC-MADDPG) algorithm. A processor configured to perform the steps of the computer-executable program, wherein the steps include: The first transceiver receives the data packets from the selected agricultural sensor storing the data packets at each time interval during the data collection process via the first link of the at least one first layer network, wherein the data collection process is performed by solving a first Markov decision process (MDP) problem based on the state of the agricultural sensor. By solving a second MDP problem formulated based on the state of the UAV agents, a route to the at least one cloud server is learned using the second transceiver via the second link of the second layer network, wherein the formulated second MDP problem is solved using multi-agent reinforcement learning (MARL), wherein the FC-MADDPG algorithm is provided to implement the MARL, wherein the execution output of the FC-MADDPG algorithm is for the route of each UAV agent; and Based on the learned route, the second transceiver is used to route the received data packets to the at least one cloud server via the second link of the second layer network.

Images