Satellite and autonomous agricultural machine network DRCO method based on deep reinforcement learning

Abstract

The invention provides a satellite and autonomous agricultural machine network DRCO method based on deep reinforcement learning, and the method comprises the steps: S1, building a satellite and autonomous agricultural machine integrated network architecture which comprises a satellite node, a ground task generation center and an autonomous agricultural machine terminal node; s2, modeling a satellite and autonomous agricultural machinery joint calculation unloading problem as a Markov decision process, defining a time-varying satellite channel gain, a calculation task and relative time for each terminal to complete the task as input variables, and constructing a sub-optimal solution optimization model for calculation unloading decision and bandwidth allocation; and S3, proposing a satellite and autonomous agricultural machine network DRCO method based on deep reinforcement learning, and generating and calculating unloading decision and bandwidth allocation scheme data by adopting an order-preserving quantization technology. According to the invention, an order-preserving quantification technology is adopted, and decision quality is ensured in a short time. The DRCO can quickly adapt to a high-randomness satellite network, and the decision performance in a dynamic environment is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing, and specifically relates to a deep reinforcement learning computation offloading method (DRCO) suitable for satellite and autonomous agricultural machinery networks. Background Art

[0002] The development of terrestrial networks is limited by the construction of ground base stations, which can only cover 6% of the world's area. The independent deployment of ground networks and satellite communication networks cannot provide global multi-dimensional resource management and continuous service capabilities. Non-terrestrial networks (NTNs) are important technologies for new application scenarios such as satellite communications and air communications in 5G networks, marking the transition from ground communications to space communications in the future.

[0003] By enabling heterogeneous agricultural machinery to be interconnected through the Internet of Things technology and combining it with edge computing technology, the current problem of agricultural labor shortage in my country can be alleviated. Agricultural machinery can offload computing tasks to the edge of the network with relatively sufficient computing resources through edge servers, which reduces the computing overhead and energy consumption of the terminal to a certain extent. However, farms are large in scope, wide in territory, and remote in location. The limited coverage area and expensive deployment cost of a single ground base station result in a low coverage rate of cellular networks in farmland. Even in some specific areas, network connectivity cannot be guaranteed, and some functions of agricultural machinery can only be used in real time when connected to a cellular network. This results in the inability of agricultural machines to perform computing tasks with high real-time requirements because they cannot receive data, further causing losses to operators. In addition, although optical fiber is faster than most network access methods, it is difficult to achieve an attractive return on investment when connecting wide-area farms because the cost of laying the necessary infrastructure is as high as $27,000 per mile. In the context of autonomous agriculture becoming a major trend in agricultural development, ground networks with low coverage and insufficient processing power cannot meet the growing demand for digitalization and intelligence in modern agriculture.

[0004] Satellite networks can effectively solve problems such as the small coverage of ground networks and can improve the output-investment ratio to a certain extent. Deploying mobile edge computing servers on satellite networks can provide reliable computing offloading services for agricultural machinery and equipment and relieve terminal computing pressure. The transmission delay of low-orbit satellites is longer than that of ground base station information transmission. If only access satellites are used as relays to ground edge computing servers, the real-time requirements of some tasks may not be met. Deploying edge computing servers on access satellites to calculate the tasks generated by agricultural machinery and equipment does not require these tasks to be transmitted over a long period of time, which is a feasible solution to the problem of lack of network coverage on farms. However, satellites need to use a considerable part of their energy and computing resources for turning and moving, so edge computing can only use a small amount of resources and storage capacity. Moreover, since the main source of satellite energy is solar energy, it is subject to strict energy consumption restrictions when it cannot receive sunlight. Satellite networks are more dynamic than ground networks, with complex network topology and routing relationships and changeable communication conditions. Compared with ground computing offloading decisions, satellite computing offloading decisions are subject to more restrictions and need to consider more factors.

[0005] At present, there are many design schemes for the satellite-ground integrated edge computing network architecture for the Internet of Vehicles, but most of them are carried out under the background of ground network coverage, which is inconsistent with the environment in which agricultural machinery is located. Although another part takes into account remote areas and satellite coverage delays, the communication characteristics of high-orbit satellites and medium-orbit satellites used do not meet the real-time requirements of agricultural machinery. Therefore, when designing the satellite-ground integrated edge computing network architecture for agricultural machinery networks, it is necessary to consider the above-mentioned limitations and take them into consideration in the subsequent algorithm design. Summary of the invention

[0006] In view of the problems that the existing computation offloading methods have insufficient adaptability in dynamic network environments, low decision-making efficiency, and high latency and energy consumption, the present invention provides a satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning.

[0007] The satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning of the present invention comprises:

[0008] S1. Establish an integrated network architecture of satellites and autonomous agricultural machinery, including satellite nodes, ground mission generation centers and autonomous agricultural machinery terminal nodes;

[0009] S2. The problem of joint computing offloading between satellite and autonomous agricultural machinery is modeled as a Markov decision process. The time-varying satellite channel gain, computing tasks, and the relative time for each terminal to complete the task are defined as input variables. The suboptimal solution optimization model for computing offloading decision and bandwidth allocation is constructed.

[0010] S3, proposes a DRCO method for satellite and autonomous agricultural machinery networks based on deep reinforcement learning, using rank-preserving quantization technology to generate computational offloading decision and bandwidth allocation scheme data;

[0011] S4. Model the time delay of offloading computing tasks to the satellite end, specifically consider the signal transmission rate based on the Shannon formula, as well as the transmission power and channel gain factors of the ground-end task generation center, establish a mathematical model of transmission delay and computing delay, and optimize the computing offloading decision based on the time delay and energy consumption of offloading computing tasks to the satellite layer;

[0012] S5. Model the total delay and energy consumption of satellite layer computing tasks, comprehensively consider the task transmission delay, computing delay and energy consumption, and the computing power of the satellite end, and optimize the algorithm to improve system performance.

[0013] Furthermore, S1 specifically includes:

[0014] S11. Establish a model of the integrated network architecture of satellites and autonomous agricultural machinery, and define the time-varying characteristics of the channel states between satellite nodes and the ground mission generation center and autonomous agricultural machinery terminal nodes;

[0015] S12. Build the task generation, communication and computing capability model of each node in the integrated network to provide basic support for subsequent computing offloading.

[0016] Furthermore, S2 specifically includes:

[0017] S21. Build a dynamic change model of satellite channel status and define channel gain, task load and calculation delay performance indicators;

[0018] S22. Model the multi-objective optimization problem as a Markov decision process, define the state space, action space and reward function, where the reward function includes the comprehensive optimization of latency, energy consumption and computing resource utilization.

[0019] Furthermore, S3 specifically includes:

[0020] S31. Use deep reinforcement learning methods combined with order-preserving quantization technology to generate suboptimal computation offloading decisions; order-preserving quantization technology is a quantization method used in deep reinforcement learning, which retains the sequence characteristics of the input data so that key information is not lost during the quantization process;

[0021] S32. Design a dynamically adjusted quantitative boundary value algorithm to prioritize offloading computing tasks to autonomous agricultural machinery terminal nodes.

[0022] Further, the S32 includes:

[0023] S321, based on the deep reinforcement learning framework, uses deep neural network to directly output computing offloading decisions and bandwidth allocation schemes;

[0024] S322: The channel gain h obtained by the satellite perception layer t , the relative task completion time tr of each computing terminal with reference to the current system time and the newly generated task sequence z t Combine them in a certain order as input and output the original floating-point vector through the deep neural network Where N represents the number of tasks; the Sigmoid function is used as the activation function of the output layer. Each element of the vector is between 0 and 1. The closer its value is to 1, the more it tends to unload the task to the specified agricultural machinery. Conversely, it tends to be calculated on the track. satisfy

[0025] S323, process the original output in an order-preserving quantization manner to obtain multiple candidate binary calculation offloading position decision vectors that meet the requirements of the quantization method The elements in this vector The specific form is

[0026] S324, using the Lagrange multiplier method to allocate the optimal bandwidth ratio to each candidate decision, calculating the cost function of each candidate decision, and selecting the decision corresponding to the minimum cost function value as the final calculation offloading solution.

[0027] Further, the S324 includes:

[0028] S3241, using candidate decision vector Calculate the bandwidth allocation scheme τ t and task offloading location x t * ;

[0029] S3242: Store the state-action pair 4-tuple of the current time frame in the experience replay buffer.

[0030] S3243. Every δ time frames, randomly draw samples from the experience replay pool, and use the randomly sampled experience data set to train the deep neural network to update the parameters θt.

[0031] Furthermore, the DNN is trained using the following loss function:

[0032]

[0033] Among them, |k t ∣ represents the sample size, T represents the transposition operation, and x t * is the actual decision value, f θt(i k ) is the network output prediction value.

[0034] Furthermore, S4 specifically includes:

[0035] First, the delay of offloading the computing task to the satellite is considered. According to Shannon's formula, the signal transmission rate of the ground-side task generation center transmission task is defined as:

[0036] R0=τ0B log2(1+p0h0 / N)

[0037] Where B is the total bandwidth, τ0 is the bandwidth ratio used by the ground-side mission generation center to transmit tasks; p0 and h0 are the transmission power of the ground-side mission generation center and the channel gain of the ground-side mission generation equipment and the satellite signal transmission path, respectively; N is the noise power; the channel gain can be calculated by We get, where t is a given time frame, is the channel gain model subject to free space path dissipation; α ti is an independent random fading factor that follows a Rice distribution, It is found by the following formula:

[0038]

[0039] Among them, A d is the antenna gain, which quantitatively describes the degree to which an antenna concentrates the input power for radiation; f c is the carrier frequency; d i is the transmission distance; d e is the path dissipation index; c is the speed of light;

[0040] Given a task size of Z i When , the transmission delay of the i-th computing task generated by the ground task generation center is as follows:

[0041]

[0042] If the task is transmitted to the satellite layer and then calculated on-orbit, the calculation delay of the task at the satellite layer is:

[0043]

[0044] Where f0 is the satellite CPU clock rate, which indicates the number of cycles the CPU can execute per second; k refers to the number of clock cycles required for the processor to process each data bit;

[0045] After defining the satellite-level latency, define the energy consumption cost by multiplying the latency by the corresponding power. Assuming that the power remains constant, the calculation method is as follows:

[0046]

[0047] Among them, p comp Calculate power for the satellite.

[0048] Furthermore, S5 specifically includes:

[0049] If the i-th computing task is offloaded to the satellite layer for on-orbit computing, the total delay to complete the calculation is:

[0050]

[0051] Accordingly, the total energy consumption from transmission to calculation completion is:

[0052]

[0053] If the task is decided to be offloaded to ground agricultural machinery for calculation, the delay and energy consumption caused by its transmission to the satellite layer are consistent with the above; the modeling of transmitting the task to agricultural machinery and calculating is:

[0054]

[0055] Among them, p trans is the satellite transmission power. The satellite layer calculates according to this parameter value when transmitting each task; τ i is the bandwidth ratio allocated to the ground layer device i; the latency and energy consumption of the ground layer device during calculation are as follows:

[0056]

[0057] Among them, p i is the calculated power of equipment i on the ground floor, f i is the corresponding CPU clock frequency.

[0058] Beneficial Effects

[0059] This paper proposes an edge intelligence architecture for satellite and autonomous agricultural machinery integrated network, and designs a computation offloading method based on deep reinforcement learning. This method is the first to study the computation offloading problem of satellite autonomous agricultural machinery network, and provides effective guidance for the design of satellite autonomous agricultural machinery network and remote computation offloading.

[0060] The technical contributions of the present invention are as follows: The present invention proposes a multi-layer Satellite and Autonomous Farm Vehicle Integrated Network (SAFVIN) architecture, and mathematically models the process of collaborative computing offloading between satellites and autonomous agricultural machinery. The computing offloading decision problem is defined as a mixed integer programming problem, while considering the continuous variable optimization problem of channel bandwidth resource allocation and the discrete variable optimization problem of computing offloading location allocation. The present invention models the satellite autonomous agricultural machinery joint computing offloading problem as a Markov decision process (MDP), providing a theoretical basis for optimization. The present invention proposes a Deep Reinforcement Computation Offloading (DRCO) method suitable for satellite networks.

[0061] Different from traditional computation offloading methods, DRCO fully considers the time-varying characteristics of satellite network channel states. This method takes the time-varying satellite channel gain, computational task requirements, and the relative time for each computing terminal to complete the task in each time frame as input, and outputs a suboptimal computation offloading decision and bandwidth allocation scheme. In addition, the present invention adopts order-preserving quantization technology to ensure the quality of decision-making in a short time. DRCO can quickly adapt to highly random satellite networks and significantly improve decision-making performance in dynamic environments. The present invention verifies the performance of the proposed DRCO through a large number of simulations.

[0062] Compared with the heuristic coordinate descent algorithm (CD), DRCO has significant advantages in latency performance. In addition, comparative experiments with other deep reinforcement learning-based algorithms show that DRCO can directly use historical experience to quickly adapt to dynamic satellite networks, while the Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) algorithms perform poorly when dealing with time-varying satellite channel gains, and the decision quality is significantly reduced. At the same time, DRCO can dynamically adjust the quantization boundary value and preferentially offload the computing tasks to the AFV node, effectively alleviating the problem of pseudo packet loss and improving system stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The satellite and autonomous agricultural machinery integrated network system architecture of the present invention;

[0064] Figure 2 It is a schematic diagram of the completion time and correlation of agricultural machinery tasks in the present invention;

[0065] Figure 3 It is the principle diagram of the DRCO method in the present invention;

[0066] Figure 4 It is a comparison chart of DRCO and CD decision performance in the present invention;

[0067] Figure 5 It is a comparison diagram of the cumulative delay of DRCO and CD in the present invention;

[0068] Figure 6 It is a schematic diagram of the decision performance of DQN in the present invention;

[0069] Figure 7 Schematic diagram of the decision performance of DDPG in the present invention;

[0070] Figure 8 It is a comparative schematic diagram of DRCO decision performance using different quantification methods in the present invention;

[0071] Fig. 9 A schematic diagram showing the impact of different cutoff values ​​on the final decision in the present invention;

[0072] Fig.10 This is a graph showing the response of the network to weight mutations in the present invention. DETAILED DESCRIPTION

[0073] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0074] like Figure 1 As shown, this embodiment builds a satellite and automatic agricultural machinery integrated network architecture consisting of a satellite layer, a ground layer, an active agricultural machinery layer and an inactive agricultural machinery layer. Among them, the satellite layer is equipped with an edge computing server, which forwards computing tasks or performs on-orbit computing of tasks based on the computing offloading decision algorithm running on the satellite. In the model of the present invention, it is considered that if a task is calculated on-orbit, it is completely calculated by the access satellite and cannot be collaboratively calculated with other satellites. The satellite layer is the core of the satellite-ground integrated network. All ground equipment within the satellite coverage area can directly or indirectly benefit from the computing offloading service through data collection, task generation and other actions. On-orbit computing can alleviate the excessive transmission time in the system and avoid unnecessary traffic between satellites and ground edge computing nodes.

[0075] The ground layer is composed of some agricultural machinery and a task generation center. These agricultural machinery include some agricultural machinery that conduct surveys on weather parameters, crop growth conditions, or soil composition, but lack computing power. These agricultural machinery cannot use their own hardware resources to quickly calculate and accurately analyze the collected information. The collected data needs to be uploaded to the task generation center, which generates tasks and then uploads them to the satellite layer. The present invention focuses on the problem of computational offloading of task sequences in a farm environment that lacks ground network coverage. Therefore, it is stipulated that the ground layer task generation center in the model can only communicate by accessing low-orbit satellites, thereby establishing contact with agricultural machinery with edge computing capabilities.

[0076] The active agricultural machinery layer is the intelligent agricultural machinery layer that is currently working in collaboration in the same farm area. It not only needs to maintain the continuous operation of its main systems, but also must ensure that various accessories related to the operation, such as sensors, navigation systems, and real-time monitoring equipment, can work stably. In addition, collaborative operations usually involve continuous operation and multitasking, which makes the equipment in the active agricultural machinery layer have stricter energy consumption restrictions, and the computing power and resources it can provide are relatively limited.

[0077] The inactive agricultural machinery layer is different from the agricultural machinery group that is performing collaborative tasks in the same farm area. These devices can be regarded as agricultural machinery that assists the active agricultural machinery layer equipment in collaborative operations in different farm areas. They usually do not face high-intensity multi-tasking requirements, and various accessories related to the operation are also turned off. Therefore, their processors and memory are not easy to reach the usage limit, making the computing power relatively sufficient. The inactive agricultural machinery layer is superior to the active agricultural machinery layer equipment and satellites mentioned above in terms of computing power and energy consumption restrictions.

[0078] In this architecture, the relevance of tasks and the time for each computing terminal device to complete the currently assigned task need to be taken into account. It is assumed that the ground layer equipment will continuously generate computing tasks in each time frame and transmit the relevant information to the satellite layer. After obtaining the task sequence size, the channel gain of each channel and the system time for each computing terminal to complete the assigned task, the satellite layer uses it as the input of the proposed algorithm and selects the algorithm output. The satellite offloads some tasks and calculates the rest on orbit.

[0079] The time for each terminal to complete the assigned task is shown as follows: Figure 2(a). First, consider the situation when the time frame is 0. At this time, each terminal can immediately calculate the newly arrived tasks. After generating the calculation offloading decision through the deep reinforcement learning method, the completion time of each task can be calculated by combining information such as channel gain and task size. First, define that the beginning of each time frame will generate a task sequence, and the transmission time of the task needs to be taken into consideration, so as to determine the time when the task arrives at the offloading target and compare it with the calculation completion time of each terminal. Therefore, when the task sequence of the next time frame arrives, it can be clearly known whether these tasks can start calculation immediately.

[0080] For the relevance of tasks, tasks in the task sequence need to be classified according to whether they have related tasks or not. For tasks without related tasks, only the calculation completion time of the unloading target needs to be considered, and then the calculation completion time of the unloading target is updated; while tasks with related tasks need to be comprehensively considered based on the completion time of the previous task and the calculation completion time of the unloading target of this task, as shown in the figure below. Figure 2 (b) as shown.

[0081] Mathematical modeling of the computational offloading process:

[0082] According to the above, it is assumed that there is only one satellite in the model. In this case, the access low-orbit satellite can cover the entire farm area and communicate with the equipment therein (including active agricultural machinery and inactive agricultural machinery) in real time. Suppose there are N active agricultural machinery and M inactive agricultural machinery in the farm range. The N active agricultural machinery equipment is represented by the set N = {1,2,...,N}, and the M inactive agricultural machinery equipment is represented by the set M = {N+1,...,N+M}. The present invention defines that in a specified farm area, data and task information need to be transmitted once from the ground layer task generation end to the satellite layer. After the satellite layer makes a task unloading decision, the task can be transmitted to the ground layer equipment via satellite transmission. At the same time, it is assumed that the channel gain in each time frame can be sensed and obtained by the satellite. The method can output the unloading location of each task and allocate bandwidth resources to each computing terminal and ground task generation center.

[0083] Calculate the offloading cost:

[0084] First, the delay of offloading the computing task to the satellite is considered. According to Shannon's formula, the signal transmission rate of the ground-side task generation center transmission task is defined as:

[0085] R0=τ0Blog2(1+p0h0 / N)

[0086] Where B is the total bandwidth, τ0 is the bandwidth ratio used by the ground-side mission generation center to transmit tasks. p0 and h0 are the transmission power of the ground-side mission generation center and the channel gain of the ground-side mission generation equipment and satellite signal transmission path, respectively, and N is the noise power. The channel gain can be calculated by We get, where t is a given time frame, It is a channel gain model that obeys free space path dissipation, which describes the power loss of electromagnetic waves during free space propagation. ti is an independent random fading factor that obeys the Rice distribution, which is a probability distribution that describes the fading phenomenon that occurs when the signal propagates through multipath in wireless communication. It can be obtained by the following formula:

[0087]

[0088] Among them, A d Antenna gain, which quantitatively describes the degree to which an antenna concentrates the input power for radiation, is an important parameter for measuring antenna directivity and efficiency; c The carrier frequency is the frequency at which the signal is transmitted. Usually, a lower signal frequency is modulated to a relatively higher frequency. This higher frequency modulated by the lower frequency is called the carrier frequency. i The transmission distance is the orbital altitude of low-orbit satellites, which is generally between 500km and 1500km; e is the path dissipation index, which is a parameter in wireless communication that describes the rate at which signal strength decreases with increasing distance; c is the speed of light. i When , the transmission delay of the i-th computing task (which can also be considered as a task that can be offloaded to the i-th agricultural machine) generated by the ground task generation center is as follows:

[0089]

[0090] If the task is transmitted to the satellite layer and then calculated on-orbit, the calculation delay of the task at the satellite layer is:

[0091]

[0092] Among them, f0 is the clock rate of the satellite CPU, which indicates the number of cycles that the CPU can execute per second and is one of the key indicators for measuring CPU performance; k refers to the number of clock cycles required for the processor to process each data bit.

[0093] After defining the satellite-level delay, the energy consumption cost can be simply defined by multiplying the delay by the corresponding power. In the present invention, it is assumed that the power is constant, and the calculation method is as follows:

[0094]

[0095] Among them, p comp Calculate power for the satellite.

[0096] According to the above analysis, if the i-th computing task is offloaded to the satellite layer for on-orbit computing, the total delay to complete the calculation is:

[0097]

[0098] Accordingly, the total energy consumption from transmission to calculation completion is:

[0099]

[0100] If the task is decided to be offloaded to ground agricultural machinery for calculation, the delay and energy consumption caused by its transmission to the satellite layer are the same as above. The modeling of transmitting the task to agricultural machinery and calculating is similar to the above, except that the parameter size has changed:

[0101]

[0102] Among them, p trans is the satellite transmission power. The present invention stipulates that the satellite layer calculates according to this parameter value when transmitting each task. i is the bandwidth ratio allocated to the ground layer device i. Similarly, the latency and energy consumption of the ground layer device when performing calculations are as follows:

[0103]

[0104] Among them, p i is the calculated power of equipment i on the ground floor, f i is the corresponding CPU clock frequency.

[0105] The principle of DRCO method is as follows Figure 3 As shown. First, the method converts the channel gain h obtained by the satellite perception layer into t , the relative task completion time tr of each computing terminal with reference to the current system time and the newly generated task sequence z t Combine them in a certain order as input and output the original floating-point vector through the deep neural network Where N represents the number of tasks. Since this method uses the Sigmoid function as the activation function of the output layer, each element of the vector is between 0 and 1. The closer its value is to 1, the more it tends to unload the task to the specified agricultural machinery. Conversely, it tends to calculate on-track. satisfy The value near 0.5 in the original output is not easy to judge its offloading tendency, so a quantization method is needed to quantize the floating point number to 0 or 1. This method uses the order-preserving quantization method to process the original output and obtain multiple candidate binary calculation offloading position decision vectors that meet the requirements of the quantization method. The elements in this vector The specific form is in After facing these quantized vectors, this method uses the Lagrange multiplier method to allocate the optimal bandwidth share to each decision, and then calculates the cost function of all candidate decisions. The candidate vector with the smallest cost function value is selected as the computation offloading decision of the task sequence in the current time frame.

[0106] Effect verification

[0107] from Figure 4 It can be seen that the delay loss caused by the decision of the DRCO method at the beginning of the experiment is worse than the optimal delay of each time frame, and the time overhead caused by the initial decision exceeds the current exhaustive decision by 5% to 10%. However, in the middle and late stages of the experiment, it can be seen that its decision in each time frame is slightly better than the optimal decision of a single time frame obtained by the exhaustive method, and the maximum is about 2% lower than the exhaustive decision. This is because the exhaustive method only considers the optimal solution within the current time frame, and the DRCO method, although it performs poorly in the early decision-making, only obtains a suboptimal solution, which also leads to different relative completion times of the calculations of each computing terminal in each subsequent time frame of the two methods. The exhaustive method maximizes the potential of each time frame, and compared with the suboptimal solution obtained by the DRCO method, it leaves less space for the next time frame, which further leads to DRCO achieving better performance in subsequent time frames.

[0108] Figure 5 The cumulative delay of all time frames before each time frame is shown. As can be seen from the figure, when the DRCO method faces a time-varying satellite network environment, the decision-making performance in the early stage will have more delay loss than the exhaustive method, but it shows an overall downward trend and has a better performance in the middle of the experiment. Compared with the exhaustive method, the decision-making running time is shorter, and the transmission and calculation delays of the task sequence are optimized, greatly reducing the delay loss. Table 1 shows the simulation parameter settings of the satellite-ground collaborative computing offloading network.

[0109] Table 1. Simulation parameter settings of satellite-ground collaborative computing offloading network

[0110] Simulation parameters Numeric Calculate the task size Z [1e+8,2e+8,...,10e+8] <![CDATA[Transmission power p0 of the ground mission generation center]]> 3dBm <![CDATA[Transmission power p of low-earth orbit satellites t > 3dBm <![CDATA[On-orbit calculated power p c > 3dBm Channel gain h Satellite network channel gain (Rician distribution) <![CDATA[Antenna gain A d > 4.11dB <![CDATA[Channel loss exponent d e > 2.8 Calculate the power p of each agricultural machine 1dBm On-orbit calculation of instruction cycle k 10Hz <![CDATA[On-orbit calculation rate f0]]> 0.5e+11cycles / s <![CDATA[Active agricultural machinery layer calculation rate f a > 0.4e+11cycles / s <![CDATA[Inactive agricultural machinery calculation rate f na > 0.6e+11cycles / s The total available bandwidth of the channel is B 800e+7bits / s Noise power N 5e-20mW

[0111] The DRCO method outperforms the CD algorithm in both performance and running time. In terms of algorithm running time overhead, the running time of DRCO is only 21.67% of that of the CD method. Therefore, the DRCO method is more suitable for collaborative satellite-ground network computing offloading environments with narrow decision windows.

[0112] Table 2 illustrates the DRCO method parameter settings.

[0113] Table 2 DRCO method parameter settings

[0114]

[0115]

[0116] Next, we test the decision performance of the DQN algorithm and the DDPG algorithm, respectively. Figure 6 as well as Figure 7 As shown. It can be seen from the figure that the DQN algorithm and the DDPG algorithm are not suitable for the model proposed by the present invention. Their average decision performance exceeds 10% of the optimal decision obtained by the exhaustive method in each time frame, and the DDPG algorithm exceeds 40%. Since the classic deep learning algorithm uses temporal difference to update the network, when the channel gain is time-varying, it cannot adapt to this strong dynamics, resulting in a slow training process and poor convergence effect. The DRCO method directly fits past suboptimal decision-making experience through experience replay technology. When the exploration of the environment is basically completed, it is possible to quickly generate computational offloading decisions that adapt to the current dynamic environment and are more adaptable to the satellite-ground collaborative network environment.

[0117] Figure 8 The performance comparison of using order-preserving quantization and binary quantization for computational offloading decisions is shown. In 10 tests of the order-preserving quantization method, the DRCO method was able to perform better than the exhaustive delay accumulation decision of each time frame. The performance of binary quantization was not good. Although there was optimization in time, there was no clear sign of convergence. Although using binary quantization can save time, it comes at the expense of algorithm performance. Comprehensive analysis shows that the order-preserving quantization method is better.

[0118] In the dynamic rank-preserving quantization method, the cutoff value L is an important factor in determining the candidate decision level. Fig. 9 As shown, the effectiveness of the optimal computing offloading decision selected by DRCO in solving the "pseudo packet loss" problem under different threshold values ​​L is explored. Since the number of tasks simulated in each round of the present invention is 10, the present invention obtains the sum of the discrete values ​​of the optimal decision in each round after adjusting the threshold value L, and divides it by 10 to constrain it between 0 and 1. According to the discrete value of the on-orbit calculation stipulated as 0 during modeling, it can be known that if the value is closer to 1, it means that more tasks are offloaded to the agricultural machinery end for calculation, thereby alleviating the calculation pressure on the satellite end. The cases where the threshold values ​​are 0.1, 0.3, 0.5, 0.7 and 0.9 are simulated here.

[0119] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning, characterized in that: The method comprises: S1. Establish an integrated network architecture of satellites and autonomous agricultural machinery, including satellite nodes, ground mission generation centers and autonomous agricultural machinery terminal nodes; S2. The problem of joint computing offloading between satellite and autonomous agricultural machinery is modeled as a Markov decision process. The time-varying satellite channel gain, computing tasks, and the relative time for each terminal to complete the task are defined as input variables. The suboptimal solution optimization model for computing offloading decision and bandwidth allocation is constructed. S3, proposes a DRCO method for satellite and autonomous agricultural machinery networks based on deep reinforcement learning, using rank-preserving quantization technology to generate computational offloading decision and bandwidth allocation scheme data; S4. Model the time delay of offloading computing tasks to the satellite end, specifically consider the signal transmission rate based on the Shannon formula, as well as the transmission power and channel gain factors of the ground-end task generation center, establish a mathematical model of transmission delay and computing delay, and optimize the computing offloading decision based on the time delay and energy consumption of offloading computing tasks to the satellite layer; S5. Model the total delay and energy consumption of satellite layer computing tasks, comprehensively consider the task transmission delay, computing delay and energy consumption, and the computing power of the satellite end, and optimize the algorithm to improve system performance.

2. The satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning according to claim 1 is characterized in that: S1 specifically includes: S11. Establish a model of the integrated network architecture of satellites and autonomous agricultural machinery, and define the time-varying characteristics of the channel states between satellite nodes and the ground mission generation center and autonomous agricultural machinery terminal nodes; S12. Build the task generation, communication and computing capability model of each node in the integrated network to provide basic support for subsequent computing offloading.

3. The satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning according to claim 1 is characterized in that: S2 specifically includes: S21. Build a dynamic change model of satellite channel status and define channel gain, task load and calculation delay performance indicators; S22. Model the multi-objective optimization problem as a Markov decision process, define the state space, action space and reward function, where the reward function includes the comprehensive optimization of latency, energy consumption and computing resource utilization.

4. The satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning according to claim 1 is characterized in that: S3 specifically includes: S31. Use deep reinforcement learning methods combined with order-preserving quantization technology to generate suboptimal computation offloading decisions; order-preserving quantization technology is a quantization method used in deep reinforcement learning, which retains the sequence characteristics of the input data so that key information is not lost during the quantization process; S32. Design a dynamically adjusted quantitative boundary value algorithm to prioritize offloading computing tasks to autonomous agricultural machinery terminal nodes.

5. The satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning according to claim 4 is characterized in that: The S32 includes: S321, based on the deep reinforcement learning framework, uses deep neural network to directly output computing offloading decisions and bandwidth allocation schemes; S322: The channel gain h obtained by the satellite perception layer t , the relative task completion time tr of each computing terminal with reference to the current system time and the newly generated task sequence z t Combine them in a certain order as input and output the original floating-point vector through the deep neural network Where N represents the number of tasks; the Sigmoid function is used as the activation function of the output layer. Each element of the vector is between 0 and 1. The closer its value is to 1, the more it tends to unload the task to the specified agricultural machinery. Conversely, it tends to be calculated on the track. satisfy S323, process the original output in an order-preserving quantization manner to obtain multiple candidate binary computation offloading position decision vectors that meet the requirements of the quantization manner The element x in this vector candii The specific form is S324, using the Lagrange multiplier method to allocate the optimal bandwidth ratio to each candidate decision, calculating the cost function of each candidate decision, and selecting the decision corresponding to the minimum cost function value as the final calculation offloading solution.

6. The satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning according to claim 5 is characterized in that: The S324 includes: S3241, using candidate decision vector Calculate the bandwidth allocation scheme τ t and task offloading location x t * ; S3242: Store the state-action pair 4-tuple of the current time frame in the experience replay buffer. S3243. Every δ time frames, randomly draw samples from the experience replay pool, and use the randomly sampled experience data set to train the deep neural network to update the parameters θt.

7. The satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning according to claim 6 is characterized in that: The DNN is trained using the following loss function: Among them, |k t ∣ represents the sample size, T represents the transposition operation, and x t * is the actual decision value, f θt (i k ) is the network output prediction value.

8. The satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning according to claim 1 is characterized in that: S4 specifically includes: First, the delay of offloading the computing task to the satellite is considered. According to Shannon's formula, the signal transmission rate of the ground-side task generation center transmission task is defined as: R0=τ0B log2(1+p0h0 / N) Where B is the total bandwidth, τ0 is the bandwidth ratio used by the ground-side mission generation center to transmit tasks; p0 and h0 are the transmission power of the ground-side mission generation center and the channel gain of the ground-side mission generation equipment and the satellite signal transmission path, respectively; N is the noise power; the channel gain can be calculated by We get, where t is a given time frame, is the channel gain model subject to free space path dissipation; α ti is an independent random fading factor that follows a Rice distribution, It is found by the following formula: Among them, A d is the antenna gain, which quantitatively describes the degree to which an antenna concentrates the input power for radiation; f c is the carrier frequency; d i is the transmission distance; d e is the path dissipation index; c is the speed of light; Given a task size of Z i When , the transmission delay of the i-th computing task generated by the ground task generation center is as follows: If the task is transmitted to the satellite layer and then calculated on-orbit, the calculation delay of the task at the satellite layer is: Where f0 is the satellite CPU clock rate, which indicates the number of cycles the CPU can execute per second; k refers to the number of clock cycles required for the processor to process each data bit; After defining the satellite-level latency, define the energy consumption cost by multiplying the latency by the corresponding power. Assuming that the power remains constant, the calculation method is as follows: Among them, p comp Calculate power for the satellite.

9. The satellite and autonomous agricultural machinery network DRCO method based on deep reinforcement learning according to claim 1 is characterized in that: S5 specifically includes: If the i-th computing task is offloaded to the satellite layer for on-orbit computing, the total delay to complete the calculation is: Accordingly, the total energy consumption from transmission to calculation completion is: If the task is decided to be offloaded to ground agricultural machinery for calculation, the delay and energy consumption caused by its transmission to the satellite layer are consistent with the above; the modeling of transmitting the task to agricultural machinery and calculating is: Among them, p trans is the satellite transmission power. The satellite layer calculates according to this parameter value when transmitting each task; τ i is the bandwidth ratio allocated to the ground layer device i; the latency and energy consumption of the ground layer device during calculation are as follows: Among them, p i is the calculated power of equipment i on the ground floor, f i is the corresponding CPU clock frequency.

Images