A communication compatibility control method

Abstract

The application discloses a communication compatibility control method, relates to the technical field of wireless communication and unmanned aerial vehicle control, and constructs a multi-unmanned aerial vehicle communication network. On a slow time scale buffer window, a bIGTOA-DMO algorithm is used to solve the discount 0-1 knapsack problem, and a content caching strategy of an unmanned aerial vehicle is obtained. On a fast time scale communication time slot, based on the content caching strategy of the unmanned aerial vehicle, an IR-MAPPO algorithm is used for dynamic decision of a multi-unmanned aerial vehicle channel, and an optimal channel selection action is output. According to the channel selection action, data transmission is performed, a total reward value containing an external reward and an internal reward is calculated, and a policy network and a value network in the IR-MAPPO algorithm are updated. The method uses an efficient algebraic evolution algorithm to solve large-scale cache combination optimization on a slow scale, uses reinforcement learning with an internal reward to solve a multi-machine cooperative communication problem on a fast scale, and realizes deep compatibility and joint control of communication and caching.

Description

Technical Field

[0001] This invention relates to the technical field of wireless communication and unmanned aerial vehicle (UAV) control, and more particularly to a communication-compatible control method. Background Technology

[0002] With the widespread application of drones in emergency communications, IoT data collection, and other fields, multi-drone collaborative communication networks face severe challenges. On the one hand, spectrum resources are scarce, and multiple drones communicating in the same area are prone to channel congestion and co-channel interference. On the other hand, the surge in service data volume overloads the backhaul link, resulting in excessively high transmission latency. Existing technologies typically treat content caching and channel access control separately. For caching decisions, precise algorithms such as dynamic programming are often used to solve the 0-1 knapsack problem, but the computational complexity is too high when dealing with massive amounts of content files, making real-time applications difficult. For channel decisions, standard multi-agent reinforcement learning is often used, but the problem of confidence allocation exists, making it difficult to distinguish the contribution of individual drones to global performance, leading to agents free-riding or getting trapped in local optima, such as all drones vying for the same channel.

[0003] Currently, Chinese invention patent application number CN202510483454.7 discloses a communication compatibility control method, device, system, and gateway. This method, applied to a gateway, specifically includes: when the gateway's main serial port cannot recognize the communication signals sent by the master device, sampling the communication signals transmitted on the receiving signal line of the main serial port through an interrupt mechanism to obtain multiple level signals; determining the baud rate of the master device based on the multiple pulse widths corresponding to the multiple level signals, and converting the multiple level signals to obtain a target bit string; parsing the target bit string to obtain the main serial port communication protocol and all byte data; decoding all byte data to obtain a string and sending it to the slave device; and configuring the main serial port according to the master device's baud rate and main serial port communication protocol. This application enables automated configuration of communication parameters between the master and slave devices through gateway bridging; and addresses the problems of low cache calculation efficiency, unclear multi-machine channel cooperation reliability allocation, and optimization of cache and communication separation in existing UAV communication control. Summary of the Invention

[0004] The technical problems solved by this invention are: low cache calculation efficiency, unclear multi-machine channel cooperation reliability allocation, and the problem of optimizing cache and communication in existing UAV communication control.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a communication compatibility control method, comprising the following steps:

[0006] Step S1: Construct a multi-UAV communication network, model the content caching decision problem as a discounted 0-1 knapsack problem, and model the channel dynamic decision problem as a partially observable Markov decision process;

[0007] Step S2: On the slow time scale cache window, use the bigtOA-DMO algorithm to solve the discounted 0-1 knapsack problem to obtain the UAV content caching strategy;

[0008] Step S3: On the fast timescale communication slot, based on the UAV content caching strategy, use the IR-MAPPO algorithm to make dynamic decisions on multiple UAV channels and output the optimal channel selection action;

[0009] Step S4: Perform data transmission according to the channel selection action, calculate the total reward value including external and internal rewards, and update the policy network and value network in the IR-MAPPO algorithm.

[0010] Preferably, step S1 includes:

[0011] The multi-UAV communication network includes a caching layer and a communication layer. The caching layer is used to map the UAV storage capacity to a discounted 0-1 knapsack problem, and the communication layer is used to map the channel access process of multiple UAVs to a partially observable Markov decision process.

[0012] The construction mapping process for the discounted 0-1 knapsack problem includes:

[0013] For each content file, define n candidate cache forms, each corresponding to one of the n item options in the itemset of the 0-1 knapsack problem, where n is a natural number greater than 2;

[0014] A caching strategy optimization objective function is established, which aims to maximize communication compatibility value, including data freshness AoI gains and backhaul bandwidth savings.

[0015] Set constraints, including that each content file can select at most one cache format, and the total size of all cached content cannot exceed the drone's cache capacity;

[0016] The specific mapping relationship between drone storage capacity and the discounted 0-1 knapsack problem includes:

[0017] In the 0-1 knapsack problem, the knapsack capacity corresponds to the upper limit of the drone's physical storage capacity;

[0018] Itemset corresponding content file;

[0019] The options within an itemset correspond to different bitrate versions of the same content file;

[0020] The item corresponds to a specific file version;

[0021] Discount constraints correspond to the grouping mutual exclusion and non-linear value superposition characteristics of content caching;

[0022] The value corresponds to the communication compatibility benefits brought by the cached content.

[0023] Preferably, the construction mapping process of the partially observable Markov decision process includes:

[0024] The objective of the partially observable Markov decision process is to maximize throughput and minimize collision probability in an environment where the channel state is time-varying and there is multi-machine interference, and to select the optimal communication channel for each UAV in real time.

[0025] The specific mapping relationships that map the channel access process of multiple UAVs to a partially observable Markov decision process include:

[0026] The intelligent agent corresponds to each drone in the network;

[0027] The environment corresponds to the wireless communication environment including all channel states, interference conditions, and physical topology;

[0028] Partial observability corresponds to the fact that UAVs cannot obtain global channel state information during actual flight. They can only obtain local observation information of received signal strength RSSI through local sensors and cannot obtain global channel state information, including the intentions of other UAVs and the global interference map.

[0029] The action corresponds to the channel number that the drone selects to access in the current time slot;

[0030] The reward corresponds to the performance feedback of the communication task, which includes data transmission rate, transmission latency, and penalty value when a channel collision occurs.

[0031] Preferably, step S2 specifically includes the following sub-steps:

[0032] Step S21: Using the DKPM2-BCode encoding method, encode the cache decision variables of the randomly generated content file into binary code. The cache decision variables include the complete cache state and the uncached state.

[0033] Step S22: Use the improved random linear combination operator MbRLCO based on inheritance strategy to perform global exploration and generate the first intermediate cache strategy individual;

[0034] Step S23: Perform local mutation on the first intermediate cache policy individual using the binary mutation operator bIRMO, specifically including the following sub-steps:

[0035] Step S231: Obtain the binary vector dimension corresponding to the first intermediate cache strategy individual, and calculate the adaptive mutation probability based on the binary vector dimension, wherein the adaptive mutation probability is the reciprocal of the binary vector dimension;

[0036] Step S232: For each binary gene in the first intermediate cache strategy individual, generate a uniformly distributed first random number independently, with the first random number in the interval [0, 1].

[0037] Step S233: Compare the first random number with the mutation probability. If the first random number is less than the mutation probability, then perform the bit-by-bit inversion operation on the binary gene corresponding to the first random number.

[0038] If the first random number is greater than or equal to the mutation probability, then the binary gene corresponding to the first random number remains unchanged;

[0039] Step S234: After traversing all 2m dimensions, a new second intermediate cache strategy individual is formed by comparing and updating all binary gene bits;

[0040] Step S24: Optimize the second intermediate cache strategy individual using the Directed Mutation Operator (DMO);

[0041] Step S25: Use the repair and optimization operator bD-GROA to process infeasible solutions, make the generated caching strategy satisfy the UAV cache capacity constraint, and output the UAV content caching strategy.

[0042] Preferably, step S22 specifically includes:

[0043] The MbRLCO operator is constructed by combining the ideas of differential evolution and inheritance strategy, and its mathematical expression is as follows:

[0044] ;

[0045] in, The i-th dimension component of the newly generated intermediate offspring gene solution is the individual with the first intermediate caching strategy. The i-th dimension of the parent gene. The second random number between [0,1] New genes generated by learning from the population;

[0046] The mathematical expression for the linear combination value generated based on other different individuals in the population is:

[0047] ;

[0048] ;

[0049] Where j is the itemset index, This represents modulo-2 addition. It is a probability discretization operator.

[0050] Preferably, step S24 specifically includes the following sub-steps:

[0051] Step S241: Calculate the value density of all content files to be decided. The value density is the ratio of the communication compatibility value of the content file to the data size. Sort all content files in descending order according to the value density and generate a priority index list. The priority index list includes the theoretical priority order of the content files to be cached.

[0052] Step S242: Based on the current second intermediate caching strategy individual, generate a third random number within the interval [0, 1], and compare the third random number with the preset directed mutation probability:

[0053] If the third random number is greater than the preset directional mutation probability, skip step S242, keep the second intermediate cache strategy individual unchanged, and proceed directly to step S25;

[0054] If the third random number is less than or equal to the preset directed mutation probability, then a directed mutation operation is triggered. The directed mutation operation includes:

[0055] Traverse the priority index list, starting with the content file with the highest value density, and check the gene state of the current second intermediate caching strategy individual. When a content file that meets the caching conditions is encountered, stop traversing and obtain the target content file. The caching conditions include: in the current caching strategy, the content file is in an uncached state, and the content file belongs to the top k in the priority index list.

[0056] Step S243: Force the corresponding gene code of the target content file to be modified to the highest value cache state.

[0057] Preferably, step S25 specifically includes the following sub-steps:

[0058] Step S251: Identify all content files marked as fully cached to form a selected content set. Calculate the total data size of all content files in the selected content set. Compare the total data size with the maximum cache capacity of the UAV. If the total data size is greater than the maximum cache capacity, it indicates that the current solution is infeasible. Perform a repair operation, which specifically includes:

[0059] Sort all content files in the selected content set from low to high value density. Starting with the file with the lowest value density, remove the files from the selected content set one by one and reset the corresponding binary code bits to the uncached state. Update the total data size of all content files in the selected content set in real time until the total data size is less than or equal to the maximum cache capacity and stop the removal operation.

[0060] Step S252: When the total data size is less than or equal to the maximum cache capacity, perform an optimization operation, which specifically includes:

[0061] Identify all currently uncached content files to form an unselected content set. Sort all content files in the unselected content set according to value density from high to low. Traverse the sorted content files and compare the data size of the current content file with the current remaining capacity. The current remaining capacity is the difference between the drone's maximum cache capacity and the total data size of the selected content set. If the data size of the current content file is less than or equal to the current remaining capacity, add the current content file to the selected content set, modify the corresponding binary code bits to the full cache state, and update the total data size and current remaining capacity of the selected content set.

[0062] If the current content file's data size exceeds the remaining capacity, then skip that content file.

[0063] The optimization operation will stop when all unselected items have been traversed or when the remaining capacity is 0.

[0064] Step S253: Output the optimal caching strategy, which is a binary caching decision vector composed of the current binary encoding of all content files, that is, the final output drone content caching strategy.

[0065] Preferably, the execution process of the IR-MAPPO algorithm includes the following sub-steps:

[0066] Step S31: Each drone, acting as an intelligent agent, independently acquires local observation information, including the received signal strength index RSSI, the cumulative interference power of the current channel, the signal-to-noise ratio, and its own buffer status.

[0067] The cache status is based on the drone content caching strategy, which checks whether the content file requested by the current user is stored in the local cache, including cache hit status and cache miss status.

[0068] Step S32: Input the local observation information into the policy network of the IR-MAPPO algorithm. The policy network is an actor network. Obtain the probability distribution of channel selection actions. The agent samples according to the probability distribution and outputs the current channel selection action. The channel selection action includes instructing the UAV to select a frequency band for data transmission in the current time slot and instructing the UAV to remain silent to avoid interference.

[0069] Step S33: All UAVs perform the corresponding channel selection action to obtain the local observation information at the next moment, reshape the reward function through the pulse adjustment mechanism, and calculate the environmental reward value based on the reshaped reward function;

[0070] The pulse modulation mechanism includes:

[0071] When a cache hit occurs and content exceeding the preset data freshness AoI threshold is successfully transmitted, a positive pulse reward is given.

[0072] When a channel collision occurs or the cumulative interference power of the channel exceeds a preset cumulative interference power threshold, a negative pulse penalty is applied.

[0073] Step S34: Combine the local observation information, channel selection action, environmental reward value, and local observation information of the next time slot corresponding to the current time slot into an experience tuple and store it in the experience replay buffer;

[0074] Step S35: Calculate the value estimation function using the critic network, estimate the advantage function based on the generalized advantage estimation method to obtain the advantage value, update the parameters of the policy network using a truncated near-end policy optimization mechanism, and obtain the optimal channel selection action. The truncated near-end policy optimization mechanism specifically includes:

[0075] The current local observation information is input into the policy network that is being updated to calculate the probability of the new policy. The ratio of the old to the new strategy is calculated using the following mathematical expression:

[0076] ;

[0077] in, The ratio of the old to the new strategies. For the probability of the new strategy, The channel selection action actually performed by the drone. For local observation information, The probability is the old strategy probability;

[0078] Construct a truncated objective function, the mathematical expression of which is:

[0079] ;

[0080] in, For a truncated objective function, As the dominant value, This is a truncation function. These are the preset hyperparameters.

[0081] Preferably, step S4 specifically includes the following sub-steps:

[0082] Step S41: Each UAV agent in the multi-UAV system performs a data transmission task in the current communication time slot according to the channel selection action output in step S3. The environment provides feedback on the communication results based on the transmission status. The communication results include the communication rate, transmission delay, channel collision status, and local observation information at the next moment.

[0083] Step S42: Based on the communication results, quantify the direct quality of the UAV completing the communication task and calculate the external reward. The mathematical expression for the external reward is:

[0084] ;

[0085] in, As an external reward, The normalized throughput for the current time slot. The normalized delay for the current transmission. This is a collision penalty term; if a channel collision occurs... =1, when no channel collision occurs =0, , and These are the weighting coefficients for throughput, latency, and collision penalty, respectively, all of which are positive real numbers;

[0086] External rewards are used to quantify the direct quality of the UAV's completion of communication tasks, reflecting the direct effects of the communication tasks, namely, encouraging high throughput and penalizing high latency and channel conflicts.

[0087] Step S43: Calculate the intrinsic reward, the calculation logic of which specifically includes:

[0088] Acquire the historical trajectory memory database, which includes the recorded channel selection action sequence and local observation information sequence of the agent within the sliding window L before the current moment;

[0089] Based on the historical trajectory memory, kernel density estimation is used to calculate the probability of the agent’s channel selection action at the current moment under the current local observation information. Based on the probability of occurrence, the Shannon entropy formula is used to calculate the trajectory spatiotemporal entropy increment brought about by the current channel selection action.

[0090] The calculated trajectory spatiotemporal entropy increment is compared with a preset novelty threshold, and a tiered reward allocation is performed. The allocation logic specifically includes:

[0091] If the trajectory spatiotemporal entropy increment is greater than or equal to the novelty threshold, the current channel selection action is determined to have high novelty, and the agent is given a positive incentive value as an intrinsic reward.

[0092] If the increase in the spatiotemporal entropy of the trajectory is less than the novelty threshold, the current channel selection action is determined to be repetitive or following historical trends, and the agent is given a basic maintenance value as an intrinsic reward.

[0093] Step S44: The external reward and the internal reward are weighted and summed to obtain the total reward value. The mathematical expression for the total reward value is:

[0094] ;

[0095] in, The total reward value, As an intrinsic reward, This is the intrinsic reward weighting coefficient;

[0096] Step S45: Store the current experience tuple into the experience replay pool, and update the policy network and value network in IR-MAPPO using the truncated near-end policy optimization algorithm.

[0097] Preferably, the cache window length is initialized and set, and step S2 is executed at the beginning of each cache window to update the drone content caching strategy and keep it unchanged;

[0098] In each communication time slot within each cache window, steps S3 and S4 are executed to dynamically adjust channel access according to the current UAV content caching strategy.

[0099] The beneficial effects of this invention are as follows: Replacing the traditional dynamic programming algorithm with the bigtOA-DMO algorithm, and significantly reducing computational complexity while improving solution speed and quality in large-scale caching problems through binary encoding, the inheritance strategy MbRLCO operator, and the directed mutation DMO operator, the invention addresses the confidence allocation problem in multi-agent cooperation by introducing an entropy-based intrinsic reward mechanism through IR-MAPPO. This incentivizes UAVs to explore different channel resources, effectively reducing channel collision rates and improving spectrum utilization. A dual-timescale architecture organically combines slow-scale caching optimization with fast-scale channel decision-making. Optimizing the caching strategy reduces backhaul load and provides better boundary conditions for channel decision-making; feedback from channel decision-making guides the evaluation of caching value, achieving end-to-end communication performance improvement. Attached Figure Description

[0100] Figure 1 This is a basic flowchart illustrating a communication compatibility control method provided in one embodiment of the present invention. Detailed Implementation

[0101] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0102] Reference Figure 1 As an embodiment of the present invention, a communication compatibility control method is provided, comprising the following steps:

[0103] Step S1: Construct a multi-UAV communication network, model the content caching decision problem as a discounted 0-1 knapsack problem, and model the channel dynamic decision problem as a partially observable Markov decision process;

[0104] Step S2: On the slow time scale cache window, use the bigtOA-DMO algorithm to solve the discounted 0-1 knapsack problem to obtain the UAV content caching strategy;

[0105] Step S3: On the fast timescale communication slot, based on the UAV content caching strategy, use the IR-MAPPO algorithm to make dynamic decisions on multiple UAV channels and output the optimal channel selection action;

[0106] Step S4: Perform data transmission according to the channel selection action, calculate the total reward value including external and internal rewards, and update the policy network and value network in the IR-MAPPO algorithm.

[0107] This method solves the large-scale cache combinatorial optimization problem at a slow scale using an efficient algebraic evolution algorithm, and solves the multi-machine collaborative communication problem at a fast scale using reinforcement learning with intrinsic rewards, achieving deep compatibility and joint control between communication and caching.

[0108] Step S1 includes:

[0109] The multi-UAV communication network includes a caching layer and a communication layer. The caching layer is used to map the UAV storage capacity to a discounted 0-1 knapsack problem, and the communication layer is used to map the channel access process of multiple UAVs to a partially observable Markov decision process.

[0110] The construction mapping process for the discounted 0-1 knapsack problem includes:

[0111] For each content file, define n candidate cache forms, each corresponding to one of the n item options in the itemset of the 0-1 knapsack problem, where n is a natural number greater than 2;

[0112] A caching strategy optimization objective function is established, which aims to maximize communication compatibility value, including data freshness AoI gains and backhaul bandwidth savings.

[0113] Set constraints, including that each content file can select at most one cache format, and the total size of all cached content cannot exceed the drone's cache capacity;

[0114] The specific mapping relationship between drone storage capacity and the discounted 0-1 knapsack problem includes:

[0115] In the 0-1 knapsack problem, the knapsack capacity corresponds to the upper limit of the drone's physical storage capacity;

[0116] Itemset corresponding content file;

[0117] The options within an itemset correspond to different bitrate versions of the same content file;

[0118] The item corresponds to a specific file version;

[0119] Discount constraints correspond to the grouping mutual exclusion and non-linear value superposition characteristics of content caching;

[0120] The value corresponds to the communication compatibility benefits brought by the cached content.

[0121] The multi-UAV communication network model is the physical carrier and system environment of the method described in this invention. This model describes a complex network system consisting of multiple UAV nodes, ground base stations, user terminals, and wireless communication links. In the multi-UAV communication network model, UAVs not only act as aerial communication relay nodes, responsible for data forwarding and transmission, but also possess a certain data storage capacity, i.e., content caching functionality. Therefore, the operational efficiency of this multi-UAV communication network model is limited by resource constraints in two core dimensions: first, the onboard storage space of UAVs is limited, making it impossible to cache massive amounts of service data; second, wireless spectrum resources are scarce and the channel environment is dynamically changing, making concurrent communication by multiple UAVs prone to co-channel interference.

[0122] Based on the constraints and characteristics of the aforementioned physical scenario, this invention decouples the resource optimization task in the multi-UAV communication network model into two interrelated engineering sub-problems: the content caching decision problem and the channel dynamic decision problem.

[0123] Content caching decision-making is a core engineering problem that needs to be solved in multi-UAV communication network models on slow timescales. Its goal is to select the most valuable subset of data from massive content files for local storage under the constraint of limited storage capacity of UAVs, so as to reduce the load on the backhaul link and improve the timeliness of data retrieval for users.

[0124] The benefit of data freshness in AoI is the sum of the product of the probability of requesting a content file and the return latency. When a user requests content, the return link transmission latency and queuing latency avoided by the drone's local cache hit are positively correlated with the popularity of the content request. If the content is cached locally on the drone, it is directly delivered by the drone when the user requests it, saving the time of transmission from the ground base station to the drone. This saved time multiplied by the probability of the file being requested is the expected improvement in AoI.

[0125] The bandwidth saving in the backhaul is the sum of the product of the request probability of the cached content and the data bitrate of the cached content. The drone responds directly to user requests through local caching, thus offloading the total data traffic from the backhaul link. If a large file has a high request probability, caching it on the drone means that backhaul bandwidth is not needed for each user request. Therefore, the file size multiplied by the request frequency is the bandwidth saving benefit brought by that file.

[0126] To mathematically solve the content caching decision problem, this invention maps it to a discounted 0-1 knapsack problem.

[0127] Encoding formats and bitrate versions include HD, SD, base layer, and enhancement layer;

[0128] The cache form refers to the specific form in which a file exists in the storage medium. It is determined by the file's encoding method and bitrate version. One encoding method and bitrate version, or a set of different encoding methods and bitrate versions, corresponds to one cache form.

[0129] The group mutual exclusion and non-linear value superposition characteristics of content caching are mapped to discount constraints. The discount constraints correspond to the group mutual exclusion and non-linear value superposition characteristics of content caching. That is, for different bitrate versions of the same content file, at most one form can be selected for caching in the final strategy.

[0130] A discounted 0-1 knapsack problem model is established with the goal of maximizing communication compatibility value and the constraint of UAV storage capacity. Through the above modeling, the content caching decision problem is transformed into a mathematical problem in the field of combinatorial optimization, which can be solved efficiently using the bigtOA-DMO algorithm proposed in this invention.

[0131] The multi-UAV communication network is refined into a caching layer (discounted 0-1 knapsack problem) and a communication layer (POMDP). Communication compatibility value (AoI benefit + bandwidth saving) is introduced as the optimization objective, and mutual exclusion constraints for multiple bitrate versions are set.

[0132] Cross-layer modeling is achieved by mapping storage capacity to backpack capacity and file version to item, which accurately establishes the correspondence between physical resources and mathematical models, making the abstract caching problem computable.

[0133] To improve overall efficiency, the proposed communication compatibility value takes into account both user experience (data freshness AoI) and network resources (backhaul bandwidth), avoiding system imbalance caused by pursuing only a single indicator;

[0134] It aligns with actual business logic and avoids redundant waste of storage resources by setting mutual exclusion constraints (each file can store at most one version), making it more consistent with the storage logic in actual engineering.

[0135] The construction mapping process of partially observable Markov decision processes includes:

[0136] The objective of the partially observable Markov decision process is to maximize throughput and minimize collision probability in an environment where the channel state is time-varying and there is multi-machine interference, and to select the optimal communication channel for each UAV in real time.

[0137] The specific mapping relationships that map the channel access process of multiple UAVs to a partially observable Markov decision process include:

[0138] The intelligent agent corresponds to each drone in the network;

[0139] The environment corresponds to the wireless communication environment including all channel states, interference conditions, and physical topology;

[0140] Partial observability corresponds to the fact that UAVs cannot obtain global channel state information during actual flight. They can only obtain local observation information of received signal strength RSSI through local sensors and cannot obtain global channel state information, including the intentions of other UAVs and the global interference map.

[0141] The action corresponds to the channel number that the drone selects to access in the current time slot;

[0142] The reward corresponds to the performance feedback of the communication task, which includes data transmission rate, transmission latency, and penalty value when a channel collision occurs.

[0143] The channel dynamic decision-making problem is a core engineering problem that needs to be solved in multi-UAV communication network models on a fast time scale. Its goal is to select the optimal communication channel for each UAV in real time under an environment where the channel state is time-varying and there is multi-UAV interference, so as to maximize system throughput and minimize collision probability.

[0144] At the communication level, the channel access process for multiple UAVs is modeled as a partially observable Markov decision process, where each UAV can only observe a local network state. Through this modeling, the channel dynamic decision problem is transformed into a mathematical problem of multi-agent sequence decision-making, enabling training and inference using the proposed IR-MAPPO algorithm to achieve distributed intelligent cooperative control.

[0145] In summary, the multi-UAV communication network model serves as the physical foundation, giving rise to optimization requirements in two dimensions: content caching and channel access. The discounted 0-1 knapsack problem and the partially observable Markov decision process are respectively the mathematical abstraction and expression of these two requirements, which together constitute the theoretical cornerstone of the communication compatibility control method of this invention.

[0146] The channel access is modeled as a partially observable Markov decision process (POMDP), which clarifies that the agent can only obtain local observation information through local sensors (RSSI), and the reward includes rate, latency and collision penalty;

[0147] To adapt to the decentralized environment and address the pain point of establishing a global communication link for drone swarms, the POMDP model is adopted, which allows drones to make decisions with only local information, thereby enhancing the robustness and scalability of the system.

[0148] Multi-objective optimization takes into account throughput, latency, and collisions simultaneously, ensuring both high efficiency and security in communication and preventing the pursuit of speed at the expense of collision risks.

[0149] Step S2 specifically includes the following sub-steps:

[0150] Step S21: Using the DKPM2-BCode encoding method, encode the cache decision variables of the randomly generated content file into binary code. The cache decision variables include the complete cache state and the uncached state.

[0151] In this embodiment, a cache update is performed at the beginning of each cache window (slow time scale);

[0152] For the discounted 0-1 knapsack problem containing n content files, the DKPM2-BCode encoding method converts the traditional m-dimensional integer encoding into a 2m-dimensional binary vector. The increased evolutionary granularity allows the algorithm to explore the solution space at a more microscopic level.

[0153] Step S22: Use the improved random linear combination operator MbRLCO based on inheritance strategy to perform global exploration and generate the first intermediate cache strategy individual;

[0154] Step S23: Perform local mutation on the first intermediate cache policy individual using the binary mutation operator bIRMO, specifically including the following sub-steps:

[0155] In this embodiment, the bIRMO operator performs random perturbation, randomly flipping certain bits in the intermediate cache strategy individuals with a small mutation probability, increasing population diversity and preventing premature convergence of the algorithm.

[0156] Step S231: Obtain the binary vector dimension corresponding to the first intermediate cache strategy individual, and calculate the adaptive mutation probability based on the binary vector dimension, wherein the adaptive mutation probability is the reciprocal of the binary vector dimension;

[0157] In this embodiment, the adaptive mutation probability is set to ensure that, statistically, each mutation operation flips only one gene in an individual vector on average, thereby achieving a small local perturbation and avoiding the destruction of the already found excellent gene structure due to excessive mutation amplitude.

[0158] Step S232: For the first intermediate cache policy individual Each binary gene in Each of the above generates a uniformly distributed first random number independently, which is within the interval [0, 1].

[0159] Step S233: Compare the first random number with the mutation probability. If the first random number is less than the mutation probability, then perform the bit-by-bit inversion operation on the binary gene corresponding to the first random number.

[0160] If the first random number is greater than or equal to the mutation probability, then the binary gene corresponding to the first random number remains unchanged;

[0161] Step S234: After traversing all 2m dimensions, a new second intermediate cache strategy individual is formed by comparing and updating all binary gene bits;

[0162] In this embodiment, the bIRMO operator performs random perturbation through the above steps, using a small mutation probability to introduce necessary randomness while maintaining the genetic stability of the population, effectively increasing population diversity and preventing the algorithm from getting trapped in local optima, i.e., premature convergence.

[0163] Step S24: Optimize the second intermediate cache strategy individual using the Directed Mutation Operator (DMO);

[0164] Step S25: Use the repair and optimization operator bD-GROA to process infeasible solutions, make the generated caching strategy satisfy the UAV cache capacity constraint, and output the UAV content caching strategy.

[0165] In this embodiment, the bD-GROA operator is a binary repair and greedy optimization operator. The bD-GROA operator first checks whether the generated caching strategy exceeds the capacity constraint. If it does, it removes the content with the lowest value density according to the greedy strategy until the constraint is met. Then, it tries to add the content with the highest value density in the remaining space to ensure that the output caching strategy is both feasible and efficient.

[0166] This method solves the large-scale cache combinatorial optimization problem at a slow scale using an efficient algebraic evolution algorithm, and solves the multi-machine collaborative communication problem at a fast scale using reinforcement learning with intrinsic rewards, achieving deep compatibility and joint control between communication and caching.

[0167] The algorithm flow of step S2 has been refined, and DKPM2-BCode encoding, MbRLCO operator, bIRMO mutation, DMO directed mutation and bD-GROA repair operator have been introduced.

[0168] By using specific binary encoding and a series of discretization operators, continuous optimization algorithms were successfully applied to discrete cache decision problems.

[0169] By combining multiple operators (MbRLCO for global exploration and DMO for targeted acceleration), the problem of traditional algorithms easily getting trapped in local optima is effectively overcome, while ensuring convergence speed.

[0170] Step S22 specifically includes:

[0171] The MbRLCO operator is constructed by combining the ideas of differential evolution and inheritance strategy, and its mathematical expression is as follows:

[0172] ;

[0173] in, The i-th dimension component of the newly generated intermediate offspring gene solution is the individual with the first intermediate caching strategy. The i-th dimension of the parent gene. The second random number between [0,1] New genes generated by learning from the population;

[0174] The mathematical expression for the linear combination value generated based on other different individuals in the population is:

[0175] ;

[0176] ;

[0177] Where j is the itemset index, This represents modulo-2 addition. It is a probabilistic discretization operator used to map linear combination values ​​in continuous space back to binary space. The specific rules include: if the product of β and the combination term exceeds a preset threshold, the corresponding bit is flipped; otherwise, it remains unchanged, thereby generating a new exploration direction.

[0178] In this embodiment, the MbRLCO operator formula represents the new solution. There is a 0.5% chance that the child will directly inherit the best traits from their parent. 0.5 probability of accepting innovative features from population interactions This achieves a balance between preserving excellent solutions and exploring new solutions;

[0179] It is randomly generated from a preset probability distribution (such as a normal distribution or a Cauchy distribution).

[0180] The preset threshold is used as the boundary for mapping the calculation results in the continuous space back to the binary space. The value is 0.5. That is, when the calculated mutation probability or combination value is greater than 0.5, it is determined that the gene has been flipped (0 becomes 1, 1 becomes 0), otherwise it remains unchanged.

[0181] An improved stochastic linear combination operator (MbRLCO) based on an inheritance strategy is defined, and the ideas of differential evolution and probability discretization are introduced.

[0182] Enhance global search capabilities by using the difference information (difference vector) between different individuals in the population to guide the generation of new individuals, enabling the algorithm to make large-scale jumps in the solution space and significantly reducing the risk of premature convergence;

[0183] Maintaining population diversity is achieved by combining random numbers with probability, ensuring the diversity of offspring genes and enabling the algorithm to find high-quality solutions even under complex constraints.

[0184] Step S24 specifically includes the following sub-steps:

[0185] Step S241: Calculate the value density of all content files to be decided. The value density is the ratio of the communication compatibility value of the content file to the data size. Sort all content files in descending order according to the value density and generate a priority index list. The priority index list includes the theoretical priority order of the content files to be cached.

[0186] It should be noted that the communication compatibility value consists of the data freshness AoI benefit and the backhaul bandwidth savings. In this embodiment, dimensional alignment and normalization are performed.

[0187] The value density calculation result does not have a specific physical unit (such as seconds / MB), but is defined as the comprehensive utility value that a unit of data storage space can provide, which serves as a unified standard for sorting and filtering.

[0188] Step S242: Based on the current second intermediate caching strategy individual, generate a third random number within the interval [0, 1], and compare the third random number with the preset directed mutation probability:

[0189] If the third random number is greater than the preset directional mutation probability, skip step S242, keep the second intermediate cache strategy individual unchanged, and proceed directly to step S25;

[0190] If the third random number is less than or equal to the preset directed mutation probability, then a directed mutation operation is triggered. The directed mutation operation includes:

[0191] Traverse the priority index list, starting with the content file with the highest value density, and check the gene state of the current second intermediate caching strategy individual. When a content file that meets the caching conditions is encountered, stop traversing and obtain the target content file. The caching conditions include: in the current caching strategy, the content file is in an uncached state, and the content file belongs to the top k in the priority index list.

[0192] Step S243: Force the corresponding gene code of the target content file to be modified to the highest value cache state.

[0193] In this embodiment, the preset directional mutation probability is set to a small value (0.05 to 0.1) to ensure that the algorithm does not completely lose its global exploration ability while using a greedy strategy to accelerate convergence.

[0194] An uncached state indicates that the corresponding binary code is 00;

[0195] k is a preset natural number, indicating that the content file is at the beginning of the priority index list and is high-quality, high-value content;

[0196] Step S243 means modifying the binary encoding of the target content file to 11. This operation is equivalent to forcibly adding the most cost-effective content that has been missed by the current strategy to the caching scheme without considering the current capacity limit.

[0197] Through the above steps, the DMO operator simulates the greedy solution process of the knapsack problem. It can quickly correct inferior individuals that have missed key high-value content in the early stages of evolution, thereby accelerating the convergence of the algorithm to high-value regions. At the same time, since this operation only occurs with a preset directional mutation probability, and the capacity will be repaired in the subsequent step S25, it will not destroy the overall diversity of the population and effectively balance development and exploration.

[0198] In this embodiment, based on the optimal solution structure characteristics of the discount 0-1 knapsack problem, it tends to select discounted items with high value density, and forces the high-value content that was not selected to be added to the cache form with a preset directional mutation probability, thereby accelerating the algorithm to converge to the high-value region; the core of step S24 is to use the directional mutation operator DMO to introduce the greedy characteristics of the knapsack problem into the evolutionary process, thereby guiding the search direction.

[0199] A directed mutation operator (DMO) is defined, which generates a priority index list based on value density and forces modification of the gene state of high-value files.

[0200] To accelerate convergence, a greedy strategy (prioritizing the caching of high-value-density files) is introduced as heuristic knowledge to guide the random search process, enabling the algorithm to quickly identify high-potential solution regions and reduce ineffective iterations.

[0201] Ensuring critical business operations and prioritizing the caching of high-value content directly improves the system's basic revenue baseline.

[0202] Step S25 specifically includes the following sub-steps:

[0203] Step S251: Identify all content files marked as fully cached to form a selected content set. Perform a mutual exclusion check on the selected content set. For the same content file, if the binary codes corresponding to multiple different bitrate versions are marked as cached at the same time, only the version with the highest value density is retained, and the binary codes corresponding to the other versions are reset to the uncached state.

[0204] The total data size of all content files in the selected content set after statistical inspection is compared with the drone's maximum cache capacity. If the total data size is greater than the maximum cache capacity, the current solution is considered infeasible, and a repair operation is performed. The repair operation specifically includes:

[0205] Sort all content files in the selected content set from low to high value density. Starting with the file with the lowest value density, remove the files from the selected content set one by one and reset the corresponding binary code bits to the uncached state. Update the total data size of all content files in the selected content set in real time until the total data size is less than or equal to the maximum cache capacity and stop the removal operation.

[0206] Step S252: When the total data size is less than or equal to the maximum cache capacity, perform an optimization operation, which specifically includes:

[0207] Identify all currently uncached content files to form an unselected content set. Sort all content files in the unselected content set according to value density from high to low. Traverse the sorted content files and compare the data size of the current content file with the current remaining capacity. The current remaining capacity is the difference between the drone's maximum cache capacity and the total data size of the selected content set. If the data size of the current content file is less than or equal to the current remaining capacity, add the current content file to the selected content set, modify the corresponding binary code bits to the full cache state, and update the total data size and current remaining capacity of the selected content set.

[0208] If the current content file's data size exceeds the remaining capacity, then skip that content file.

[0209] The optimization operation will stop when all unselected items have been traversed or when the remaining capacity is 0.

[0210] Step S253: Output the optimal caching strategy, which is a binary caching decision vector composed of the current binary encoding of all content files, that is, the final output drone content caching strategy.

[0211] In step S251, when the binary code is 11, it means that the content file is fully cached, and when the binary code is 00, it means that the content file is not cached. The current solution is the second intermediate caching strategy individual after being processed by the directional mutation operator DMO, which ensures that high-value content is retained as much as possible while satisfying the constraints.

[0212] After the above repair and optimization operations, the generated caching strategy vector is a valid solution that satisfies the constraints and maximizes value under the current constraints. This vector is output as the optimal caching strategy for this iteration and used for subsequent fitness evaluation or updating the next generation of the population.

[0213] The repair and optimization operator bD-GROA is defined, which includes removing low-value repair operations and filling high-value optimization operations;

[0214] To ensure the feasibility of the solution, when the algorithm generates an infeasible solution that exceeds the capacity through random search, it can be automatically corrected to a feasible solution that satisfies the constraints, ensuring that the output strategy is physically executable.

[0215] By maximizing resource utilization and filling unselected high-value files with remaining space while meeting constraints, storage space is not wasted, and the value of cache capacity is further squeezed.

[0216] The execution process of the IR-MAPPO algorithm includes the following sub-steps:

[0217] Step S31: Each drone, acting as an intelligent agent, independently acquires local observation information, including the received signal strength index RSSI, the cumulative interference power of the current channel, the signal-to-noise ratio, and its own buffer status.

[0218] The cache status is based on the drone content caching strategy, which checks whether the content file requested by the current user is stored in the local cache, including cache hit status and cache miss status.

[0219] During the training phase, the central controller collects local observation information and joint actions of all UAVs to construct a global state vector for the evaluation of the critic network's value.

[0220] The cache hit status and cache miss status correspond to the complete cache status and the uncached status mentioned above, respectively.

[0221] Step S32: Input the local observation information into the policy network of the IR-MAPPO algorithm. The policy network is an actor network. Obtain the probability distribution of channel selection actions. The agent samples according to the probability distribution and outputs the current channel selection action. The channel selection action includes instructing the UAV to select a frequency band for data transmission in the current time slot and instructing the UAV to remain silent to avoid interference.

[0222] Step S33: All UAVs perform the corresponding channel selection action to obtain the local observation information at the next moment, reshape the reward function through the pulse adjustment mechanism, and calculate the environmental reward value based on the reshaped reward function;

[0223] The pulse modulation mechanism includes:

[0224] When a cache hit occurs and content exceeding the preset data freshness AoI threshold is successfully transmitted, a positive pulse reward is given.

[0225] When a channel collision occurs or the cumulative interference power of the channel exceeds a preset cumulative interference power threshold, a negative pulse penalty is applied.

[0226] The mathematical expression for the reshaped reward function is:

[0227] ;

[0228] in, Let be the environmental reward value obtained by the drone at time t. It is a positive pulse excitation factor. and As a negative impulse penalty factor, As a collision penalty factor, As a interference penalty factor, For indicator functions, This indicates whether the content file requested by the current user is stored in the local cache. In transmission status, The data freshness AoI of the currently transmitted content. The preset data freshness threshold, For channel collision indication, This represents the cumulative interference power of the current channel. This is the preset cumulative interference power threshold;

[0229] The positive pulse excitation factor is a positive real number, representing the preset reward intensity given to the agent when all high-quality transmission conditions are met;

[0230] The negative pulse penalty factors are all negative real numbers, representing the preset penalty intensity given to the agent when a communication failure or adverse environment occurs;

[0231] Both the positive impulse excitation factor and the negative impulse penalty factor are manually set, and the value of α in the reward function is 2.0. The value is 10.0. The value is set to 5.0 to ensure that the drone learns to avoid collisions during the initial training phase.

[0232] Indicator functions include: the function value is 1 when the condition inside the parentheses is true, and the function value is 0 when the condition inside the parentheses is false;

[0233] A value of 1 indicates a cache hit. A value of 0 indicates a cache miss.

[0234] A value of 1 indicates that the data packet was successfully decoded. A value of 0 indicates that the data packet transmission failed;

[0235] A high reward is only given when the transmitted content is old enough, i.e., has a high AoI and urgently needs to be updated; if the content is very new, the reward for transmitting it is not so high, i.e. it is filtered out by the impulse mechanism.

[0236] A value of 1 indicates that a multi-machine collision has occurred in the current channel. A value of 0 indicates no conflict;

[0237] In the normalization model of this embodiment, the preset data freshness threshold is set as a dimensionless integer, such as 10 to 20 time slots.

[0238] The preset cumulative interference power threshold is derived from the hardware sensitivity of the wireless communication receiver or the lower limit of the signal-to-interference-plus-noise ratio in the Shannon formula. When the interference power exceeds this value, the channel quality deteriorates, resulting in the inability to properly demodulate the signal and communication interruption. This value needs to be calculated by back-calculating the minimum SINR required by the actual modulation and demodulation method (such as QPSK, 16QAM).

[0239] When the interference exceeds the preset cumulative interference power threshold, the environment is judged as unusable and extremely congested.

[0240] Step S34: Combine the local observation information, channel selection action, environmental reward value, and local observation information of the next time slot corresponding to the current time slot into an experience tuple and store it in the experience replay buffer;

[0241] Step S35: Calculate the value estimation function using the critic network, estimate the advantage function based on the generalized advantage estimation method to obtain the advantage value, update the parameters of the policy network using a truncated near-end policy optimization mechanism, and obtain the optimal channel selection action. The truncated near-end policy optimization mechanism specifically includes:

[0242] The current local observation information is input into the policy network that is being updated to calculate the probability of the new policy. The ratio of the old to the new strategy is calculated using the following mathematical expression:

[0243] ;

[0244] in, The ratio of the old to the new strategies. For the probability of the new strategy, The channel selection action actually performed by the drone. For local observation information, The probability is the old strategy probability;

[0245] Construct a truncated objective function, the mathematical expression of which is:

[0246] ;

[0247] in, For a truncated objective function, As the dominant value, This is a truncation function. These are the preset hyperparameters.

[0248] In this embodiment, the critic network is the value network;

[0249] The hyperparameter is set to 0.2, which means that the ratio can only be a maximum of 1.2 and a minimum of 0.8 to prevent the policy from updating too rapidly.

[0250] if > 1 indicates that the new strategy is more likely to perform this action than the old strategy;

[0251] if < 1 indicates that the new strategy reduces the probability of performing this action;

[0252] By maximizing this objective function and updating the policy network parameters θ using the gradient ascent method, the optimal channel access policy is obtained.

[0253] After multiple rounds of such iterative updates, the policy network The network will gradually converge, and the final network will be able to output the channel access action with the optimal probability distribution based on the current channel interference and buffer state, thereby maximizing the cumulative expected total reward (defined by the impulse reward function) and minimizing collisions and interference while ensuring data freshness AoI.

[0254] To enhance the algorithm's responsiveness to high-value, sudden tasks, an impulse adjustment mechanism is introduced to reshape the reward function;

[0255] By maximizing the objective function and updating the policy network parameters, the optimal channel access policy is obtained by maximizing the cumulative expected total reward while ensuring the stability of policy updates.

[0256] Step S35 aims to update the Actor network (policy network) using the previously calculated advantage function, while using a truncation mechanism to prevent excessively large network parameter updates from causing training failure.

[0257] The estimation of the dominance function based on the generalized dominance estimation method to obtain the dominance value includes: calculating the time series difference error, and recursively calculating the dominance value based on the time series difference error;

[0258] Before performing the update, a batch of training data had been collected from the previous steps. For each time step t, there is a corresponding current state, the channel selection action actually performed by the UAV, the dominance value, and the old policy probability. The current state is the UAV's local observation information, and the policy probability is the probability that the network will choose that action when collecting data.

[0259] The IR-MAPPO algorithm is a proximal strategy optimization algorithm for impulse-modulated multi-agent systems.

[0260] After determining the optimal caching strategy for the current large time period in step S2, the communication time slot loop with a fast time scale is entered. In each communication time slot, channel selection and power control for multi-UAV collaboration are realized.

[0261] The IR-MAPPO algorithm process has been refined, especially by introducing a pulse adjustment mechanism (positive / negative pulse) and truncated PPO update;

[0262] The impulse regulation mechanism is similar to a stimulant, providing high-intensity feedback when critical events (such as successful transmission of data in a cache hit or a collision) occur, helping the neural network to quickly identify key good and bad actions and shorten the training cycle.

[0263] To improve training stability, truncated PPO limits the update range between the old and new policies, preventing policy collapse caused by an excessively large update step size, and ensuring the convergence stability of the algorithm in dynamic environments.

[0264] Cross-layer collaboration incorporates its own buffer state into the observation state, enabling channel selection to be aware of the buffer status (e.g., a more aggressive transmission strategy may be adopted when the buffer is hit), thus achieving linkage between communication and buffering.

[0265] Step S4 specifically includes the following sub-steps:

[0266] Step S41: Each UAV agent in the multi-UAV system performs a data transmission task in the current communication time slot according to the channel selection action output in step S3. The environment provides feedback on the communication results based on the transmission status. The communication results include the communication rate, transmission delay, channel collision status, and local observation information at the next moment.

[0267] Step S42: Based on the communication results, quantify the direct quality of the UAV completing the communication task and calculate the external reward. The mathematical expression for the external reward is:

[0268] ;

[0269] in, As an external reward, The normalized throughput for the current time slot. The normalized delay for the current transmission. This is a collision penalty term; if a channel collision occurs... =1, when no channel collision occurs =0, , and These are the weighting coefficients for throughput, latency, and collision penalty, respectively, all of which are positive real numbers;

[0270] The weighting coefficients for throughput, latency, and collision penalty are summed to 1 to balance the optimization objective of high throughput with low collision rate.

[0271] External rewards are used to quantify the direct quality of the UAV's completion of communication tasks, reflecting the direct effects of the communication tasks, namely, encouraging high throughput and penalizing high latency and channel conflicts.

[0272] Step S43: Calculate the intrinsic reward, the calculation logic of which specifically includes:

[0273] Acquire the historical trajectory memory database, which includes the recorded channel selection action sequence and local observation information sequence of the agent within the sliding window L before the current moment;

[0274] L is the preset sliding window length (e.g., L=1000), and the kernel density is estimated using the Gaussian kernel function;

[0275] Based on the historical trajectory memory, kernel density estimation is used to calculate the probability of the agent’s channel selection action at the current moment under the current local observation information. Based on the probability of occurrence, the Shannon entropy formula is used to calculate the trajectory spatiotemporal entropy increment brought about by the current channel selection action.

[0276] The mathematical expression for the increment of the spacetime entropy of the trajectory is given by the following formula:

[0277] ;

[0278] in, For the trajectory spatiotemporal entropy increment, Let $\mathbf{ ... To prevent extremely small positive numbers whose logarithms are meaningless (e.g., 1e-6);

[0279] The trajectory spatiotemporal entropy increment quantifies the novelty of the current action relative to historical behavior patterns. The lower the probability of occurrence, the less common the action is in history, and the larger the trajectory spatiotemporal entropy increment value.

[0280] The calculated trajectory spatiotemporal entropy increment is compared with a preset novelty threshold (set to 1.5), and a tiered reward allocation is performed. The allocation logic specifically includes:

[0281] If the trajectory spatiotemporal entropy increment is greater than or equal to the novelty threshold, the current channel selection action is determined to have high novelty, and the agent is given a positive incentive value as an intrinsic reward.

[0282] If the increase in the spatiotemporal entropy of the trajectory is less than the novelty threshold, the current channel selection action is determined to be repetitive or following historical trends, and the agent is given a basic maintenance value as an intrinsic reward.

[0283] In this embodiment, the determination that the current channel selection action has a high novelty representation leads to a high entropy increase in the system state, and the positive incentive value Set to the preset maximum intrinsic reward constant, for example =1;

[0284] In this embodiment, the baseline maintenance value Set to a preset minimum intrinsic reward constant, for example A value of 0 is intended to prevent the agent from receiving excessively high false rewards when novel exploration is lacking, thus forcing it to seek new strategies to obtain rewards. .

[0285] Intrinsic rewards are used to quantify the exploratory value of agent behavior based on trajectory spatiotemporal entropy;

[0286] This mechanism is used to solve the confidence allocation problem in multi-agent collaboration. By rewarding drones that make unique contributions or explore unknown spectrum states, it prevents agents from falling into a lazy state or local optimum due to the inability to distinguish individual contributions.

[0287] Step S44: The external reward and the internal reward are weighted and summed to obtain the total reward value. The mathematical expression for the total reward value is:

[0288] ;

[0289] in, The total reward value, As an intrinsic reward, This is the intrinsic reward weighting coefficient;

[0290] The total reward value is the ultimate target value for agent optimization in reinforcement learning, and it is also the direct basis for the critic network to conduct value evaluation. The virtual reward introduced to solve the problem of multi-agent cooperation, the intrinsic reward represents the exploratory value of the action and its contribution to the increase of system entropy;

[0291] This is a preset hyperparameter (with a value between 0.01 and 0.1) used to balance the importance of external task objectives (exploitation) and intrinsic exploration motivation (exploration). The larger the value, the more the algorithm encourages the drone to try unknown channel strategies;

[0292] Step S45: Store the current experience tuple into the experience replay pool, and update the policy network and value network in IR-MAPPO using the truncated near-end policy optimization algorithm.

[0293] In this embodiment, by maximizing the cumulative expected total reward, the UAV learns a channel access strategy that can both meet the communication quality requirements guided by external rewards and maintain good collaboration and exploration capabilities guided by internal rewards.

[0294] If an agent's channel selection action results in high entropy increase or high novelty in the system state, a higher intrinsic reward value is assigned to incentivize the agent to explore unoccupied channel resources and solve the credibility allocation problem in multi-agent cooperation.

[0295] A total reward function was designed that includes both external and intrinsic rewards. The intrinsic reward, based on a historical trajectory memory bank and Shannon entropy, is used to measure the novelty of an action.

[0296] When the complex communication environment makes it difficult to obtain or change external rewards (such as throughput), intrinsic rewards can drive the agent to actively explore the unknown state-action space.

[0297] To avoid suboptimal convergence, rewarding novel actions encourages drones to try different channels and strategies, preventing agents from sticking to a safe but inefficient channel and thus discovering globally better communication strategies.

[0298] By adapting to dynamic environments and using a sliding window L to focus on recent trajectories, intrinsic rewards can adapt to dynamic changes in the environment rather than being constrained by outdated historical data.

[0299] Initialize and set the cache window length. At the beginning of each cache window, execute step S2 to update the drone content caching strategy and keep it unchanged.

[0300] In each communication time slot within each cache window, steps S3 and S4 are executed to dynamically adjust channel access according to the current UAV content caching strategy, forming a closed-loop collaboration between cache optimization and channel decision-making.

[0301] The execution logic is defined by time scale, the caching strategy is updated and maintained at the beginning of the window, and the channel strategy is dynamically adjusted in each time slot.

[0302] By reducing computational overhead and performing highly complex combinatorial optimization problems (caching) on ​​a slower time scale, the computational load on the system is reduced, avoiding heavy computation in every time slot.

[0303] To ensure system real-time performance, communication decisions (channel selection) are executed on a fast time scale, enabling real-time response to millisecond-level channel fading and interference changes, thus ensuring communication reliability.

[0304] The operation boundaries of the two time scales are clearly defined, which allows the caching algorithm and the communication algorithm to be developed in parallel and optimized independently, making it easier for the system to be implemented and maintained in engineering.

[0305] This invention replaces the traditional dynamic programming algorithm with the bigtOA-DMO algorithm. Through binary encoding, the inheritance strategy MbRLCO operator, and the directed mutation DMO operator, it significantly reduces computational complexity and improves solution speed and quality in large-scale caching problems. By introducing an entropy-based intrinsic reward mechanism through IR-MAPPO, it solves the confidence allocation problem in multi-agent cooperation, incentivizing UAVs to explore different channel resources, effectively reducing channel collision rate and improving spectrum utilization. A dual-timescale architecture is adopted, organically combining the slow-scale caching optimization with the fast-scale channel decision-making. Optimizing the caching strategy reduces backhaul load and provides better boundary conditions for channel decision-making; feedback from channel decision-making guides the evaluation of caching value, achieving end-to-end communication performance improvement.

[0306] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0307] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A communication compatibility control method, characterized in that, Includes the following steps: Step S1: Construct a multi-UAV communication network, model the content caching decision problem as a discounted 0-1 knapsack problem, and model the channel dynamic decision problem as a partially observable Markov decision process; Step S2: On the slow time scale cache window, use the bigtOA-DMO algorithm to solve the discounted 0-1 knapsack problem to obtain the UAV content caching strategy; Step S3: On the fast timescale communication slot, based on the UAV content caching strategy, use the IR-MAPPO algorithm to make dynamic decisions on multiple UAV channels and output the optimal channel selection action; Step S4: Perform data transmission according to the channel selection action, calculate the total reward value including external and internal rewards, and update the policy network and value network in the IR-MAPPO algorithm.

2. The communication compatibility control method as described in claim 1, characterized in that, Step S1 includes: The multi-UAV communication network includes a caching layer and a communication layer. The caching layer is used to map the UAV storage capacity to a discounted 0-1 knapsack problem, and the communication layer is used to map the channel access process of multiple UAVs to a partially observable Markov decision process. The construction mapping process for the discounted 0-1 knapsack problem includes: For each content file, define n candidate cache forms, each corresponding to one of the n item options in the itemset of the 0-1 knapsack problem, where n is a natural number greater than 2; A caching strategy optimization objective function is established, which aims to maximize communication compatibility value, including data freshness AoI gains and backhaul bandwidth savings. Set constraints, including that each content file can select at most one cache format, and the total size of all cached content cannot exceed the drone's cache capacity; The specific mapping relationship between drone storage capacity and the discounted 0-1 knapsack problem includes: In the 0-1 knapsack problem, the knapsack capacity corresponds to the upper limit of the drone's physical storage capacity; Itemset corresponding content file; The options within an itemset correspond to different bitrate versions of the same content file; The item corresponds to a specific file version; Discount constraints correspond to the grouping mutual exclusion and non-linear value superposition characteristics of content caching; The value corresponds to the communication compatibility benefits brought by the cached content.

3. The communication compatibility control method as described in claim 2, characterized in that, The construction mapping process of the partially observable Markov decision process includes: The objective of the partially observable Markov decision process is to maximize throughput and minimize collision probability in an environment where the channel state is time-varying and there is multi-machine interference, and to select the optimal communication channel for each UAV in real time. The specific mapping relationships that map the channel access process of multiple UAVs to a partially observable Markov decision process include: The intelligent agent corresponds to each drone in the network; The environment corresponds to the wireless communication environment including all channel states, interference conditions, and physical topology; Partial observability corresponds to the fact that UAVs cannot obtain global channel state information during actual flight. They can only obtain local observation information of received signal strength RSSI through local sensors and cannot obtain global channel state information, including the intentions of other UAVs and the global interference map. The action corresponds to the channel number that the drone selects to access in the current time slot; The reward corresponds to the performance feedback of the communication task, which includes data transmission rate, transmission latency, and penalty value when a channel collision occurs.

4. The communication compatibility control method as described in claim 1, characterized in that, Step S2 specifically includes the following sub-steps: Step S21: Using the DKPM2-BCode encoding method, encode the cache decision variables of the randomly generated content file into binary code. The cache decision variables include the complete cache state and the uncached state. Step S22: Use the improved random linear combination operator MbRLCO based on inheritance strategy to perform global exploration and generate the first intermediate cache strategy individual; Step S23: Perform local mutation on the first intermediate cache policy individual using the binary mutation operator bIRMO, specifically including the following sub-steps: Step S231: Obtain the binary vector dimension corresponding to the first intermediate cache strategy individual, and calculate the adaptive mutation probability based on the binary vector dimension, wherein the adaptive mutation probability is the reciprocal of the binary vector dimension; Step S232: For each binary gene in the first intermediate cache strategy individual, generate a uniformly distributed first random number independently, with the first random number in the interval [0, 1]. Step S233: Compare the first random number with the mutation probability. If the first random number is less than the mutation probability, then perform the bit-by-bit inversion operation on the binary gene corresponding to the first random number. If the first random number is greater than or equal to the mutation probability, then the binary gene corresponding to the first random number remains unchanged; Step S234: After traversing all 2m dimensions, a new second intermediate cache strategy individual is formed by comparing and updating all binary gene bits; Step S24: Optimize the second intermediate cache strategy individual using the Directed Mutation Operator (DMO); Step S25: Use the repair and optimization operator bD-GROA to process infeasible solutions, make the generated caching strategy satisfy the UAV cache capacity constraint, and output the UAV content caching strategy.

5. The communication compatibility control method as described in claim 4, characterized in that, Step S22 specifically includes: The MbRLCO operator is constructed by combining the ideas of differential evolution and inheritance strategy, and its mathematical expression is as follows: ； in, The i-th dimension component of the newly generated intermediate offspring gene solution is the individual with the first intermediate caching strategy. The i-th dimension of the parent gene. The second random number between [0,1] New genes generated through learning from the population; The mathematical expression for the linear combination value generated based on other different individuals in the population is: ； ； Where j is the itemset index, This represents modulo-2 addition. It is a probability discretization operator.

6. The communication compatibility control method as described in claim 4, characterized in that, Step S24 specifically includes the following sub-steps: Step S241: Calculate the value density of all content files to be decided. The value density is the ratio of the communication compatibility value of the content file to the data size. Sort all content files in descending order according to the value density and generate a priority index list. The priority index list includes the theoretical priority order of the content files to be cached. Step S242: Based on the current second intermediate caching strategy individual, generate a third random number within the interval [0, 1], and compare the third random number with the preset directed mutation probability: If the third random number is greater than the preset directional mutation probability, skip step S242, keep the second intermediate cache strategy individual unchanged, and proceed directly to step S25; If the third random number is less than or equal to the preset directed mutation probability, then a directed mutation operation is triggered. The directed mutation operation includes: Traverse the priority index list, starting with the content file with the highest value density, and check the gene state of the current second intermediate caching strategy individual. When a content file that meets the caching conditions is encountered, stop traversing and obtain the target content file. The caching conditions include: in the current caching strategy, the content file is in an uncached state, and the content file belongs to the top k in the priority index list. Step S243: Force the corresponding gene code of the target content file to be modified to the highest value cache state.

7. The communication compatibility control method as described in claim 4, characterized in that, Step S25 specifically includes the following sub-steps: Step S251: Identify all content files marked as fully cached to form a selected content set. Calculate the total data size of all content files in the selected content set. Compare the total data size with the maximum cache capacity of the UAV. If the total data size is greater than the maximum cache capacity, it indicates that the current solution is infeasible. Perform a repair operation, which specifically includes: Sort all content files in the selected content set from low to high value density. Starting with the file with the lowest value density, remove the files from the selected content set one by one and reset the corresponding binary code bits to the uncached state. Update the total data size of all content files in the selected content set in real time until the total data size is less than or equal to the maximum cache capacity and stop the removal operation. Step S252: When the total data size is less than or equal to the maximum cache capacity, perform an optimization operation, which specifically includes: Identify all currently uncached content files to form an unselected content set. Sort all content files in the unselected content set according to value density from high to low. Traverse the sorted content files and compare the data size of the current content file with the current remaining capacity. The current remaining capacity is the difference between the drone's maximum cache capacity and the total data size of the selected content set. If the data size of the current content file is less than or equal to the current remaining capacity, add the current content file to the selected content set, modify the corresponding binary code bits to the full cache state, and update the total data size and current remaining capacity of the selected content set. If the current content file's data size exceeds the remaining capacity, then skip that content file. The optimization operation will stop when all unselected items have been traversed or when the remaining capacity is 0. Step S253: Output the optimal caching strategy, which is a binary caching decision vector composed of the current binary encoding of all content files, that is, the final output drone content caching strategy.

8. The communication compatibility control method as described in claim 1, characterized in that, The execution process of the IR-MAPPO algorithm includes the following sub-steps: Step S31: Each drone, acting as an intelligent agent, independently acquires local observation information, including the received signal strength index RSSI, the cumulative interference power of the current channel, the signal-to-noise ratio, and its own buffer status. The cache status is based on the drone content caching strategy, which checks whether the content file requested by the current user is stored in the local cache, including cache hit status and cache miss status. Step S32: Input the local observation information into the policy network of the IR-MAPPO algorithm. The policy network is an actor network. Obtain the probability distribution of channel selection actions. The agent samples according to the probability distribution and outputs the current channel selection action. The channel selection action includes instructing the UAV to select a frequency band for data transmission in the current time slot and instructing the UAV to remain silent to avoid interference. Step S33: All UAVs perform the corresponding channel selection action to obtain the local observation information at the next moment, reshape the reward function through the pulse adjustment mechanism, and calculate the environmental reward value based on the reshaped reward function; The pulse modulation mechanism includes: When a cache hit occurs and content exceeding the preset data freshness AoI threshold is successfully transmitted, a positive pulse reward is given. When a channel collision occurs or the cumulative interference power of the channel exceeds a preset cumulative interference power threshold, a negative pulse penalty is applied. Step S34: Combine the local observation information, channel selection action, environmental reward value, and local observation information of the next time slot corresponding to the current time slot into an experience tuple and store it in the experience replay buffer; Step S35: Calculate the value estimation function using the critic network, estimate the advantage function based on the generalized advantage estimation method to obtain the advantage value, update the parameters of the policy network using a truncated near-end policy optimization mechanism, and obtain the optimal channel selection action. The truncated near-end policy optimization mechanism specifically includes: The current local observation information is input into the policy network that is being updated to calculate the probability of the new policy. The ratio of the old to the new strategy is calculated using the following mathematical expression: ； in, The ratio of the old to the new strategies. For the probability of the new strategy, The channel selection action actually performed by the drone. For local observation information, The probability is the old strategy probability; Construct a truncated objective function, the mathematical expression of which is: ； in, For a truncated objective function, As the dominant value, This is a truncation function. These are the preset hyperparameters.

9. The communication compatibility control method as described in claim 1, characterized in that, Step S4 specifically includes the following sub-steps: Step S41: Each UAV agent in the multi-UAV system performs a data transmission task in the current communication time slot according to the channel selection action output in step S3. The environment provides feedback on the communication results based on the transmission status. The communication results include the communication rate, transmission delay, channel collision status, and local observation information at the next moment. Step S42: Based on the communication results, quantify the direct quality of the UAV completing the communication task and calculate the external reward. The mathematical expression for the external reward is: ； in, As an external reward, The normalized throughput for the current time slot. The normalized delay of the current transmission. This is a collision penalty term; if a channel collision occurs... =1, when no channel collision occurs. =0, , and These are the weighting coefficients for throughput, latency, and collision penalty, respectively, all of which are positive real numbers; External rewards are used to quantify the direct quality of the UAV's completion of communication tasks, reflecting the direct effects of the communication tasks, namely, encouraging high throughput and penalizing high latency and channel conflicts. Step S43: Calculate the intrinsic reward, the calculation logic of which specifically includes: Acquire the historical trajectory memory database, which includes the recorded channel selection action sequence and local observation information sequence of the agent within the sliding window L before the current moment; Based on the historical trajectory memory, kernel density estimation is used to calculate the probability of the agent’s channel selection action at the current moment under the current local observation information. Based on the probability of occurrence, the Shannon entropy formula is used to calculate the trajectory spatiotemporal entropy increment brought about by the current channel selection action. The calculated trajectory spatiotemporal entropy increment is compared with a preset novelty threshold, and a tiered reward allocation is performed. The allocation logic specifically includes: If the trajectory spatiotemporal entropy increment is greater than or equal to the novelty threshold, the current channel selection action is determined to have high novelty, and the agent is given a positive incentive value as an intrinsic reward. If the trajectory spatiotemporal entropy increment is less than the novelty threshold, the current channel selection action is determined to be repetitive or follows the historical trend, and the agent is given a basic maintenance value as an intrinsic reward. Step S44: The external reward and the internal reward are weighted and summed to obtain the total reward value. The mathematical expression for the total reward value is: ； in, The total reward value, As an intrinsic reward, This is the intrinsic reward weighting coefficient; Step S45: Store the current experience tuple into the experience replay pool, and update the policy network and value network in IR-MAPPO using the truncated near-end policy optimization algorithm.

10. The communication compatibility control method as described in claim 1, characterized in that: Initialize and set the cache window length. At the beginning of each cache window, execute step S2 to update the drone content caching strategy and keep it unchanged. In each communication time slot within each cache window, steps S3 and S4 are executed to dynamically adjust channel access according to the current UAV content caching strategy.

Images