Multi-agent collaborative data collection learning method based on alliance formation game

Abstract

The invention discloses a multi-agent collaborative data collection learning method based on an alliance formation game. The method comprises the following steps: constructing a communication scene of an unmanned vehicle and a ground heterogeneous sensor in an industrial environment; establishing a communication link model of the unmanned vehicle, the heterogeneous sensor and the data collection center; constructing a dynamic data model of the unmanned vehicle and the heterogeneous sensor; designing a relative fairness evaluation index of the heterogeneous sensor; establishing an unmanned vehicle trajectory optimization model; a heterogeneous sensor alliance is designed to form a game model, and it is proved that Nash equilibrium exists to deal with the complex association problem of the unmanned vehicle and heterogeneous sensors; designing a sensor association alliance forming algorithm; and standardizing the optimization model as a partially observable Markov decision process, and designing a multi-agent collaborative learning trajectory optimization method in combination with an alliance structure to obtain an optimal unmanned vehicle path plan. The data collection fairness and efficiency can be improved, the system throughput is improved, and the safety and stability of task execution of the unmanned vehicle are ensured.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication confrontation technology, and in particular to a multi-agent collaborative data collection and learning method based on alliance formation game. Background Art

[0002] In recent years, in the development of the Industrial Internet of Things, the collaborative work of unmanned vehicles and heterogeneous sensors has attracted much attention. As a mobile data collection platform, unmanned vehicles are highly dependent on widely distributed and dispersed sensors in the environment to obtain rich environmental information. However, the wireless transmission of sensors often leads to unstable data transmission due to factors such as signal attenuation and interference, while the deployment of wired links faces problems such as high cost and difficulty in construction, which greatly limits the data collection channels. In addition, the complex relationship between unmanned vehicles and heterogeneous sensors makes effective data collection and communication challenging.

[0003] In the communication scenario between unmanned vehicles and heterogeneous sensors, multiple factors are intertwined, making the data interaction problem extremely complicated. For example, obstacles in the environment will hinder signal transmission and affect the communication efficiency between unmanned vehicles and sensors; the geographical location of the data collection center will also affect signal propagation and data transmission rate, causing delays and data packet loss.

[0004] In terms of fairness, the phenomenon of unbalanced resource allocation is common. Some sensors may collect data frequently due to the access preferences of unmanned vehicles, while other sensors may not collect data for a long time and are idle or overloaded. This unfair resource allocation not only affects the overall performance of the system, but may also lead to a decrease in the timeliness and accuracy of information acquisition, thereby affecting the stability and reliability of the industrial monitoring system.

[0005] In terms of efficiency, traditional methods have difficulty in effectively handling complex association problems. The associations between unmanned vehicles and sensors are complex and changeable, and traditional algorithms have difficulty adapting to such changes quickly, resulting in low data collection efficiency. At the same time, the problem of limited data transmission also seriously restricts the throughput of the system and cannot meet the needs of large-scale data transmission in industrial scenarios.

[0006] Most existing technologies are single-agent optimizations, focusing on improving the performance of unmanned vehicles or single sensors, but failing to fully consider the interactive characteristics between unmanned vehicles and sensors. Although some studies have improved the efficiency of path planning and task scheduling for unmanned vehicles through optimization algorithms, there is still a lack of holistic thinking about the collaborative work of multiple agents. For example, during the data collection process, if the data acquisition capabilities of sensors and changes in environmental conditions are ignored, the unmanned vehicle may face problems of inefficiency and information redundancy when collecting data.

[0007] In general, the current collaborative data collection methods for unmanned vehicles and heterogeneous sensors mainly face the following four problems: 1) Insufficient adaptability to complex correlations: Existing methods lack effective strategies when dealing with complex dynamic correlations between unmanned vehicles and heterogeneous sensors, making it difficult to achieve collaborative data collection. 2) Deficiencies in resource allocation fairness and data transmission efficiency: Current solutions are prone to unfairness in the resource allocation process, affecting the access frequency and data transmission efficiency of sensors, and thus affecting the performance of the overall system. 3) Lack of systematic thinking on the collaborative work of multiple agents: Existing research focuses on the performance improvement of a single agent, lacks a comprehensive analysis of the collaboration mechanism between multiple agents, and limits the overall optimization capabilities of the system. Summary of the invention

[0008] In industrial scenarios, sensors face many challenges in wireless data transmission due to power limitations, and traditional data collection methods have deficiencies in fairness, efficiency, and processing of complex associations. In response to these difficulties in sensor data collection in existing industrial scenarios, the present invention provides a multi-agent collaborative data collection learning method based on alliance formation game, which can significantly improve the fairness and efficiency of data collection, ensure the safety and stability of unmanned vehicles when performing tasks, and effectively handle the complex association problems between unmanned vehicles and heterogeneous sensors.

[0009] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0010] A multi-agent collaborative data collection and learning method based on alliance formation game, the multi-agent collaborative data collection and learning method comprising the following steps:

[0011] Step S1, constructing a communication scenario between an unmanned vehicle and ground heterogeneous sensors in an industrial environment;

[0012] Step S2, establishing a communication link model between the unmanned vehicle and the ground heterogeneous sensors and data collection center;

[0013] Step S3, constructing dynamic data models of the unmanned vehicle and ground heterogeneous sensors respectively;

[0014] Step S4, designing a relative fairness evaluation index for ground heterogeneous sensors;

[0015] Step S5, maximizing the system throughput by considering the association status of ground heterogeneous sensors with the unmanned vehicle and the position of the unmanned vehicle, and establishing an unmanned vehicle trajectory optimization model in the data collection scenario in combination with collision constraints and power constraints;

[0016] Step S6, designing a coalition formation game model for ground heterogeneous sensors, formulating exchange criteria and insertion criteria, dealing with the complex association problem between the unmanned vehicle and ground heterogeneous sensors on the basis of maximizing system throughput, and using precise potential energy game to prove the existence of Nash equilibrium;

[0017] Step S7, designing a ground heterogeneous sensor alliance formation algorithm, combining the exchange criterion and the insertion criterion, and outputting a stable alliance structure;

[0018] Step S8, converting the unmanned vehicle trajectory optimization model established in step S5 into a partially observable Markov decision process, combining the alliance structure output in step S7 to design a multi-agent collaborative learning trajectory optimization method to obtain the optimal unmanned vehicle path planning.

[0019] Step S1 further comprises:

[0020] Set up a simulation scenario, which includes a data collection center, N unmanned vehicles, M ground heterogeneous sensors and multiple obstacles The set of unmanned vehicles is represented as N = {1, 2..., N}, the set of ground heterogeneous sensors is represented as M = {1, 2, .., M}, and the available channels of unmanned vehicles are represented as Each unmanned vehicle independently selects a ground heterogeneous sensor alliance and channel to collect ground heterogeneous sensor data. D offloading data to a data collection center;

[0021] The duration of the unmanned vehicle data collection is discretized into T equal time slots Δτ, and the position coordinates of the unmanned vehicle n in the tth time slot are It is expressed as:

[0022]

[0023]

[0024]

[0025] Where t = {1, 2, ..., T}, v represents the moving speed of the unmanned vehicle, d represents the displacement per unit time, represents the deflection angle of the unmanned vehicle n at the tth time slot; The following conditions are met:

[0026]

[0027]

[0028] Where, d ugv,min represents the minimum collision distance between unmanned vehicles, d ob,minrepresents the minimum collision distance between the unmanned vehicle and the obstacle. and are the unmanned vehicle n′ and the obstacle Position coordinates at the tth time slot.

[0029] Step S2 further comprises:

[0030] Considering the impact of obstacles on the communication link, the channel gain between the unmanned vehicle n and the ground heterogeneous sensor m is expressed as:

[0031]

[0032] in, represents the distance between the unmanned vehicle n and the ground heterogeneous sensor m in the tth time slot, α is the path loss constant, β is the average channel gain; η represents the shadow effect and follows a Gaussian distribution The subscripts Los and NLos denote line-of-sight and non-line-of-sight, respectively;

[0033] Assume that the communication between the unmanned vehicle and the ground sensor adopts the time division multiple access mode. In each time slot, the unmanned vehicle can only collect data from one ground heterogeneous sensor. The signal received by the unmanned vehicle n from the ground sensor m in the tth time slot is:

[0034]

[0035] Among them, P m represents the transmission power of the ground heterogeneous sensor m, represents the interference signal within the alliance, σ 2 is the white Gaussian noise power that affects the communication link between the unmanned vehicle and the sensor; the data transmission rate from the ground heterogeneous sensor m to the unmanned vehicle n in the tth time slot is: c n It represents the bandwidth of the data collected by drone n;

[0036] The data unloading rate from the unmanned vehicle to the data collection center is defined as:

[0037]

[0038] in, Indicates that other unmanned vehicles n' are also on channel B D The interference caused by transmitting data on the m represents the transmission power of unmanned vehicle n, represents the white noise power when the unmanned vehicle unloads data, represents the channel gain from unmanned vehicle n to the data collection center.

[0039] Step S3 further comprises:

[0040] Constructing a dynamic model of the data of ground heterogeneous sensor m at time slot t+1 for:

[0041]

[0042] Among them, v m is the speed at which the ground heterogeneous sensor m collects data from the environment, is the amount of sensing data collected by the ground heterogeneous sensor m by the unmanned vehicle n in time slot t, is the association state between the unmanned vehicle n and the ground heterogeneous sensor m at time t, D m,max represents the data storage capacity of the ground heterogeneous sensor m;

[0043] Construct a dynamic data model of the unmanned vehicle n at time slot t+1 for:

[0044]

[0045] in, represents the amount of data unloaded by unmanned vehicle n to the data collection center at time t, D n,max Represents the data storage capacity of unmanned vehicle n.

[0046] In step S4, the Jain fairness index is introduced as an indicator to measure the fairness of ground heterogeneous sensor data transmission:

[0047]

[0048] in, Indicates relative collection amount; represents the amount of data sent by the ground heterogeneous sensor m to the unmanned vehicle n during a long-term mission; It represents the total amount of data collected by the ground heterogeneous sensor m from the environment before the current time slot t.

[0049] In step S5, the unmanned vehicle trajectory optimization model in the data collection scenario is:

[0050]

[0051]

[0052] In the formula, is the amount of sensing data collected by the ground heterogeneous sensor m by the unmanned vehicle n in time slot t, is the association state between the unmanned vehicle n and the ground heterogeneous sensor m at time t, l t is the position of the unmanned vehicle at time t, d ugv,min represents the minimum collision distance between unmanned vehicles, is the position coordinate of the unmanned vehicle n′ at the tth time slot, f t As an indicator to measure the fairness of ground heterogeneous sensor data transmission, represents the remaining power of the unmanned vehicle n in time slot t, and B represents the location set of obstacles.

[0053] Step S6 further comprises:

[0054] S61, let the ground heterogeneous sensors be participants and regard the unmanned vehicle-heterogeneous sensor association problem as a non-cooperative game Where M represents the set of participants, {S m} m∈M represents the strategy set of the participants, {u m} m∈M Denotes the utility function of the participant, and defines the utility function of the ground heterogeneous sensor m as the relative amount of data collection:

[0055]

[0056] in, represents the data transmission rate between the ground heterogeneous sensor m and the unmanned vehicle n at time slot t, represents the total amount of data collected from the environment by the ground heterogeneous sensor m to the current time slot t; then the alliance C i The profit function is:

[0057] S62, consider maximizing the benefits of the alliance to design the alliance exchange criteria and joining criteria, where:

[0058] The exchange criterion is: if the ground heterogeneous sensor s1∈C i , s2∈C j , when the ground heterogeneous sensors s1, s2 meet the following conditions, the alliance exchange operation is performed:

[0059] The joining criterion is: if the ground heterogeneous sensor s1∈C i , when the ground heterogeneous sensor s1 meets the following conditions, it leaves the alliance C i Join Alliance C j :

[0060] Among them, δ m represents the strategy of sensor m, δ -m represents the sensor association strategy except sensor m;

[0061] S63, using exact potential energy game to prove the existence of Nash equilibrium.

[0062] Step S7 further comprises:

[0063] S71, randomly generate a set of unmanned vehicle position coordinates The coordinate set of ground heterogeneous sensors l m And obstacle collection Initialize the ground heterogeneous sensors and unmanned vehicle transmission power P m ,P n and bandwidth aggregation

[0064] S72, calculating the distance from each ground heterogeneous sensor to each unmanned vehicle, and randomly selecting unmanned vehicles to form an initial alliance;

[0065] S73, randomly select a ground heterogeneous sensor s1 and obtain the alliance C where the ground heterogeneous sensor s1 currently belongs i ;

[0066] S74, randomly select an alliance C j ,C i ≠C j , in C j A ground heterogeneous sensor s2 is randomly selected;

[0067] S75, Computing Alliance C i and C j The utility function and

[0068] S76, determine whether the alliance update criteria are met, and if the alliance exchange criteria are met, update the alliance C i ←{C i / s1}∪s2; C j ←{C j / s2}∪s1; if the alliance joining criteria are met, update alliance C j ←C j ∪s1;C i ←C i / s1;

[0069] S77, repeating steps S73 to S76, when the number of repetitions reaches the maximum number of iterations, outputting a stable alliance structure {C i} i∈N .

[0070] Step S8 further comprises:

[0071] S81, the unmanned vehicle trajectory optimization model is standardized as a partially observable Markov decision process, and the partially observable Markov decision process is represented as a tuple (o t ,s t ,a t ,o t+1 ,st+1 ,r t ), where o represents the observation space, s represents the global state space, a represents the action space, and the reward function r represents the evaluation of the current observation action pair (o, a);

[0072] S82, combined with the alliance structure, designs a trajectory optimization method based on multi-agent collaborative learning. The design process includes the following steps:

[0073] S821, determine the model parameters of the optimizer Adam, which include learning rate α, discount factor γ, strategy clipping range ε ​​and experience pool Initialize the Actor network parameters π of each unmanned vehicle n,θ and Critic network parameters V n,θ ;

[0074] S822, initialize the observation space of each unmanned vehicle

[0075] S823, obtain the association between the unmanned vehicle and heterogeneous sensors

[0076] S824, using the Actor strategy network based on the current observation results Output probability distribution Sampling actions from probability distributions and the corresponding probability Set the selected action Input into the environment to obtain the observation space for the next time slot And calculate the reward of the current state based on the feedback from the environment

[0077] S825, use the Critic network to calculate the value of the current action

[0078] S826, repeat steps S822 to S825 to obtain the current observation, global state, action, action probability, reward, next observation and global state of each unmanned vehicle (o t ,s t ,a t ,π t ,r t ,o t+1 ,s t+1 ) ; Store some observable Markov decision processes into the experience pool

[0079] S827, update model parameters, process batch data, calculate advantage function, perform strategy pruning, and maintain the stability of the update strategy;

[0080] S828, repeating steps S823 to S827 until all unmanned vehicles run out of power or the mission is completed;

[0081] S829: When the maximum number of training times is reached, the optimal unmanned vehicle trajectory is output

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] First, the multi-agent collaborative data collection learning method based on alliance formation game of the present invention effectively avoids the sensor overload problem caused by uneven resource allocation by designing the sensor relative fairness evaluation index, enhances the stability and reliability of the industrial monitoring system, and ensures the comprehensiveness of data collection.

[0084] Second, the multi-agent collaborative data collection learning method based on the coalition formation game of the present invention uses the coalition formation game strategy to handle the complex association state between the unmanned vehicle (UAV) and the sensor, maximizes the system throughput, solves the problem of limited sensor data transmission, and realizes efficient data collection.

[0085] Third, the multi-agent collaborative data collection and learning method based on alliance formation game of the present invention takes into account collision constraints and power constraints to ensure that each unmanned vehicle maintains a safe distance and avoids obstacles when performing tasks to avoid collisions. A partially observable Markov decision process is designed. The unmanned vehicle can still maintain efficient decision-making when facing uncertainties such as communication fluctuations and environmental changes, further enhancing the robustness and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 A diagram of a scenario in which the unmanned vehicle designed for the present invention collects data from ground sensors.

[0087] Figure 2 Schematic diagram of the alliance exchange criteria and insertion criteria designed for the present invention.

[0088] Figure 3 This is a framework diagram of the multi-agent trajectory planning algorithm of the present invention.

[0089] Figure 4 A comparison chart of the alliance formation algorithm designed for the present invention and other algorithms.

[0090] Figure 5 This is a comparison chart of the convergence performance of the algorithm of the present invention and different multi-agent deep reinforcement learning trajectory optimization methods.

[0091] Figure 6 This is a comparison chart of the convergence performance of different unmanned vehicle-sensor association algorithms used in the trajectory planning algorithm designed for the present invention. DETAILED DESCRIPTION

[0092] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0093] The present invention discloses a multi-agent collaborative data collection and learning method based on alliance formation game, and the multi-agent collaborative data collection and learning method comprises the following steps:

[0094] Step S1, constructing a communication scenario between an unmanned vehicle and ground heterogeneous sensors in an industrial environment;

[0095] Step S2, establishing a communication link model between the unmanned vehicle and the ground heterogeneous sensors and data collection center;

[0096] Step S3, constructing dynamic data models of the unmanned vehicle and ground heterogeneous sensors respectively;

[0097] Step S4, designing a relative fairness evaluation index for ground heterogeneous sensors;

[0098] Step S5, maximizing the system throughput by considering the association status of ground heterogeneous sensors with the unmanned vehicle and the position of the unmanned vehicle, and establishing an unmanned vehicle trajectory optimization model in the data collection scenario in combination with collision constraints and power constraints;

[0099] Step S6, designing a coalition formation game model for ground heterogeneous sensors, formulating exchange criteria and insertion criteria, dealing with the complex association problem between the unmanned vehicle and ground heterogeneous sensors on the basis of maximizing system throughput, and using precise potential energy game to prove the existence of Nash equilibrium;

[0100] Step S7, designing a ground heterogeneous sensor alliance formation algorithm, combining the exchange criterion and the insertion criterion, and outputting a stable alliance structure;

[0101] Step S8, converting the unmanned vehicle trajectory optimization model established in step S5 into a partially observable Markov decision process, combining the alliance structure output in step S7 to design a multi-agent collaborative learning trajectory optimization method to obtain the optimal unmanned vehicle path planning.

[0102] The multi-agent collaborative data collection and learning method of the present invention specifically includes the following steps:

[0103] In step S1, a communication scenario between an unmanned vehicle and heterogeneous ground sensor equipment in an industrial environment is constructed.

[0104] like Figure 1 As shown in the simulation scenario, there is a data collection center (DC), N unmanned vehicles, M sensors and multiple obstacles. The set of unmanned vehicles is represented by N = {1, 2..., N}, and the set of ground sensors is represented by M = {1, 2, .., M}. The available channels for unmanned vehicles can be represented as Each unmanned vehicle independently selects a ground sensor alliance and channel to collect sensor data. D The data is unloaded to the data collection center. In addition, the duration of the data collected by the unmanned vehicle is discretized into T equal time slots Δτ, then t={1,2,...,T}. Therefore, the position coordinates of the unmanned vehicle n in time slot t are It is expressed as:

[0105]

[0106]

[0107]

[0108] Among them, v represents the moving speed of the unmanned vehicle, and θ∈[0,2π] represents the deflection angle of the unmanned vehicle. In addition, when performing tasks, each unmanned vehicle should maintain a safe distance and avoid obstacles to avoid collisions in time slot t:

[0109]

[0110]

[0111] Among them, d ugv,min represents the minimum collision distance between unmanned vehicles, d ob,min Indicates the minimum collision distance between the unmanned vehicle and the obstacle.

[0112] In step S2, the communication link models between the heterogeneous sensors and the unmanned vehicle, and between the unmanned vehicle and the data collection center are determined.

[0113] Considering the impact of obstacles on the communication link, the channel gain between the unmanned vehicle n and the sensor m is expressed as:

[0114]

[0115] in, represents the distance between UAV n and sensor m, α is the path loss constant, β is the average channel gain, and η represents the shadow effect, which follows a Gaussian distribution.

[0116] Assume that the communication between the unmanned vehicle and the ground heterogeneous sensors adopts the time division multiple access mode. The rule is that in each time slot, the unmanned vehicle can only collect data from one sensor, and the signal SINR received from the ground equipment is as follows:

[0117]

[0118] Among them, P m represents the transmission power of sensor m, represents the interference signal within the alliance. Therefore, the data transmission rate from sensor m to unmanned vehicle n is:

[0119] Similarly, define the data unloading rate from the unmanned vehicle to the data collection center:

[0120]

[0121] in, Indicates that other unmanned vehicles n' are also on channel B D interference caused by transmitting data on the

[0122] In step S3, a dynamic data model of the unmanned vehicle and heterogeneous sensors is constructed. Specifically, the following steps are included:

[0123] Step S31, constructing a dynamic data model of heterogeneous sensors.

[0124] Initially, the sensor m moves at a constant speed v m To generate data, the unmanned vehicle n needs to collect some sensor data from m and then unload it to the data collection center. Therefore, the data stored by sensor m and unmanned vehicle n are dynamically updated. For sensor m, its data dynamic model at time slot t can be expressed as:

[0125]

[0126] Among them, v m is the speed at which sensor m collects data from the environment, It is defined as the amount of sensing data collected by sensor m by unmanned vehicle n in time slot t, Defined as the association state between the unmanned vehicle n and the sensor m at time t, D m,max represents the data storage capacity of sensor m, and we require that each unmanned vehicle is connected to at most one sensor in each time slot.

[0127] Step S32, constructing a dynamic data model of the unmanned vehicle.

[0128] For unmanned vehicle n, the data at time t includes the amount of data collected from the sensor alliance and the amount of data unloaded to the data collection center. The dynamic data model of unmanned vehicle n is defined as:

[0129]

[0130] in, represents the amount of data unloaded from the unmanned vehicle-mx to the data collection center at time t, D n,max Dm,max Represents the data storage capacity of unmanned vehicle n.

[0131] Step S4, designing a relative fairness evaluation index for heterogeneous sensors.

[0132] In order to achieve fair performance of data collection between heterogeneous sensors and maximize the total data collection of the system, the Jain fairness index is introduced as an indicator to measure the fairness of sensor data transmission:

[0133]

[0134] in, Relative collection amount, Represents the amount of data sent by sensors to the unmanned vehicle during a long mission; It represents the sum of the data collected by the sensor from the environment before the current time slot t. By optimizing the Jain fairness index, we can improve the fairness between sensors while ensuring the system throughput, improve the stability and reliability of the entire industrial monitoring system, and avoid affecting the accuracy of monitoring results due to the backlog of some sensor data.

[0135] Step S5, establishing an unmanned vehicle trajectory optimization model in a data collection scenario.

[0136] Based on the fairness index and dynamic data model, our goal is to consider the heterogeneous sensors associated with the autonomous vehicle. and the position of the unmanned vehicle l t To maximize the throughput of the system. Therefore, considering the collision constraints C3, C5 and the power constraint C4, the optimization objective can be expressed as:

[0137]

[0138]

[0139] Step S6, design a game model for forming a heterogeneous sensor alliance, formulate exchange criteria and insertion criteria, handle the complex association problems between unmanned vehicles and heterogeneous sensors on the basis of maximizing system throughput, and use precise potential energy game to prove the existence of Nash equilibrium.

[0140] Step S61: We regard the unmanned vehicle-heterogeneous sensor association problem as a non-cooperative game Where M represents the set of participants, i.e., heterogeneous sensors, {S m} m∈M represents the strategy set of the participants, {u m} m∈MRepresents the utility function of the participant. Our optimization goal is to maximize the system throughput while ensuring the fairness of sensor offloading data. Therefore, we define the utility function of sensor m as the relative data collection amount:

[0141]

[0142] in, represents the data transmission rate between sensor m and unmanned vehicle n at time slot t, represents the total amount of data collected from the environment by sensor m to the current time slot t. Then alliance C i The profit function is:

[0143]

[0144] Step S62, as Figure 2 As shown, considering maximizing the benefits of the alliance, the alliance exchange criteria and joining criteria are designed:

[0145] Exchange criterion: If sensor s1∈C i , s2∈C j , when sensors s1 and s2 meet the following conditions, the alliance exchange operation is performed:

[0146] Joining criterion: If sensor s1∈C i , when sensor s1 meets the following conditions, it leaves alliance C i Join Alliance C j :

[0147] Step S63, using the exact potential energy game to prove the existence of Nash equilibrium.

[0148] According to the definition of exact potential game: when a game It is called an exact potential game, then there exists a potential energy function Φ, for any participant m∈M, the following equation holds:

[0149]

[0150] According to the theorem: an exact potential game has at least one pure strategy Nash equilibrium. The potential function of the coalition formation game is defined as the sum of the utilities of all coalitions:

[0151]

[0152] Among them, the decision of the mth sensor node is δ m , and U i (δ m ,δ _m ) means C i The utility of alliances.

[0153] When sensor m changes its decision from δ n Change to When , the change of potential function can be calculated as:

[0154]

[0155] Among them, Δ k Indicates the addition of C i ,C j The changes in the utility function of other alliances other than Δ are independent of the entry or exit actions of sensor node m, so Δ k =0.

[0156] In addition, the change in node utility caused by the decision change of sensor m is:

[0157]

[0158] This proves that the game It is an exact potential game. For any sensor m, change its strategy δ independently. m The change in the coalition utility function is the same as the change in the potential energy function, so there is at least one pure strategy Nash equilibrium solution.

[0159] Step S7, designing an alliance formation algorithm. The algorithm specifically includes the following steps:

[0160] Step S71: Randomly generate a set of unmanned vehicle position coordinates The coordinate set of ground heterogeneous sensors l m And the obstacle set Initialize the transmission power P of heterogeneous sensors and unmanned vehicles m ,P n , bandwidth set

[0161] Step S72, calculating the distance from each sensor to each unmanned vehicle, and randomly selecting unmanned vehicles to form an initial alliance;

[0162] Step S73, randomly select a sensor s1 and obtain the alliance C where s1 currently belongs. i ;

[0163] Step S74, randomly select an alliance C j (C j ≠C i ), in C j A sensor s2 is randomly selected;

[0164] Step S75, calculate alliance C i and C j The utility function

[0165] Step S76: determine whether the alliance update criteria are met. If the alliance exchange criteria are met, update the alliance C. i ←{C i / s1}∪s2; C j ←{C j / s2}∪s1; if the alliance joining criteria are met, update alliance C j ←C j ∪s1;C i ←C i / s1;

[0166] Step S77, repeating steps S73 to S76, when the number of repetitions reaches the maximum number of iterations, outputting a stable alliance structure {C i} i∈N .

[0167] Step S8: normalize the optimization model designed in step S5 into a partially observable Markov decision process, combine it with the alliance structure obtained in step S7, design a trajectory optimization method based on multi-agent collaborative learning, and obtain the optimal unmanned vehicle path planning. Figure 3 As shown, the specific steps include:

[0168] Step S81, normalize the optimization model into a partially observable Markov decision process, which can be represented as a tuple (o t ,s t ,a t ,o t+1 ,s t+1 ,r t ), where o represents the observation space, s represents the global state space, a represents the action space, and the reward function r represents the evaluation of the current observation action pair (o, a). in represents the location set of all unmanned vehicles; Represents the location set of sensors in the unmanned vehicle alliance; They represent the distance set between unmanned vehicle n and other unmanned vehicles, the distance set between n and sensors in the alliance, and the distance set between unmanned vehicle n and obstacles. Represents the remaining data set of sensors in the alliance; action space is the deflection angle of the unmanned vehicle n; the global state space s=(o1,o2,...o N ) contains all the observation information of the unmanned vehicle; the reward function in Indicates power penalty. represents the collision penalty, where E(·) represents the indicator function; represents the data collection reward, Indicates a data offloading reward.

[0169] Step S82, combining the alliance structure obtained in S7, designing a trajectory optimization method based on multi-agent collaborative learning:

[0170] Step S821, determine the model parameters of the optimizer Adam, which include learning rate α, discount factor γ, strategy clipping range ε ​​and experience pool Initialize the Actor network parameters π of each unmanned vehicle n,θ and Critic network parameters V n,θ ;

[0171] Step S822: Initialize the observation space of each unmanned vehicle

[0172] Step S823, according to S7, the association between the unmanned vehicle and the heterogeneous sensor is obtained

[0173] Step S824, using the Actor strategy network based on the current observation results Output probability distribution Actions sampled from a probability distribution and the corresponding probability Set the selected action Input into the environment to obtain the observation space for the next time slot And calculate the reward of the current state based on the feedback from the environment

[0174] Step S825, use the Critic network to calculate the value of the current action

[0175] Step S826, repeat steps S822-S825 to obtain the current observation, global state, action, action probability, reward, next observation and global state of each unmanned vehicle (o t ,s t ,a t ,π t ,r t ,o t+1 ,s t+1 ); store some observable Markov decision processes into the experience pool.

[0176] Step S827, update model parameters, process batch data, and calculate advantage function Update Actor Network Parameters Update Critic network parameters like or Then the strategy is pruned to maintain the stability of the update strategy;

[0177] Step S828, repeat steps S823-S827 until all unmanned vehicles run out of power or complete the mission;

[0178] Step S829: When the maximum number of training times is reached, the optimal unmanned vehicle trajectory is output.

[0179] Table 1 shows the computational complexity of different algorithms in the scenario of the present invention. Where N and M represent the number of unmanned vehicles and the number of sensors respectively; D u represents the sensor data collected by the unmanned vehicle n, D B represents the data unloaded by the unmanned vehicle to the data collection center; O represents the observation dimension of a single unmanned vehicle; C1, C2, and C3 represent the computational complexity of the Optimal, BR, and CFG algorithms, respectively:

[0180] Table 1 Computational complexity of different algorithms.

[0181]

[0182]

[0183] Figure 4 A comparison chart of the alliance formation algorithm designed by the present invention and other algorithms is given. It can be observed that the Optimal algorithm shows the fastest convergence speed, followed by the BR algorithm. Although the convergence process of the CFG algorithm is slow, it can achieve the optimal solution and reach a stable state after multiple iterations. This is mainly because Optimal and BR are centralized algorithms that rely on global information. When there are a large number of participants, the computational complexity increases significantly. In contrast, the CFG algorithm shows excellent performance advantages in complex dynamic environments by optimizing information interaction and computational complexity.

[0184] Figure 5 The convergence performance of the present invention is compared with that of multi-agent deep reinforcement learning methods such as MAPPO and QMIX. Although MAPPO and QMIX have advantages in computational complexity, their convergence performance is significantly inferior to that of MAPPO using the CFG algorithm when considering system fairness. The present invention achieves an effective balance between fairness and system throughput, providing a better solution for multi-agent trajectory optimization.

[0185] Figure 6The following is a comparison chart of the convergence performance of different association algorithms used in the present invention. Although the Optimal and BR algorithms show faster convergence speeds in the process of determining user associations, the CFG algorithm performs better in terms of the effect of optimizing system goals. This is because CFG fully considers the principle of fairness and can effectively balance the interests of all parties while optimizing the goals, thereby improving the overall system performance.

[0186] In summary, the present invention proposes a multi-agent collaborative data collection learning method based on alliance formation game. By designing the sensor relative fairness evaluation index, the sensor overload problem caused by uneven resource allocation is avoided, the stability and reliability of the industrial monitoring system are enhanced, and comprehensive data collection is ensured. The alliance formation game strategy is used to handle the complex relationship between unmanned vehicles and sensors, maximize the system throughput, solve the problem of limited data transmission, and achieve efficient data collection. Considering collision constraints and power constraints, ensure that the unmanned vehicle can perform tasks safely and avoid collisions. The optimization model is standardized as a partially observable Markov decision process, so that the unmanned vehicle can still make efficient decisions in the face of uncertainty, and enhance the robustness and stability of the system. The designed sensor alliance formation algorithm and multi-agent collaborative learning trajectory optimization method show good performance in the comparison of different algorithms, achieve an effective balance between fairness and system throughput, and provide a better solution for unmanned vehicle trajectory optimization. The good characteristics of the present invention are proved by simulation.

[0187] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0188] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0191] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0192] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multi-agent collaborative data collection and learning method based on alliance formation game, characterized in that: The multi-agent collaborative data collection and learning method comprises the following steps: Step S1, constructing a communication scenario between an unmanned vehicle and ground heterogeneous sensors in an industrial environment; Step S2, establishing a communication link model between the unmanned vehicle and the ground heterogeneous sensors and data collection center; Step S3, constructing dynamic data models of the unmanned vehicle and ground heterogeneous sensors respectively; Step S4, designing a relative fairness evaluation index for ground heterogeneous sensors; Step S5, maximizing the system throughput by considering the association status of ground heterogeneous sensors with the unmanned vehicle and the position of the unmanned vehicle, and establishing an unmanned vehicle trajectory optimization model in the data collection scenario in combination with collision constraints and power constraints; Step S6, designing a coalition formation game model for ground heterogeneous sensors, formulating exchange criteria and insertion criteria, dealing with the complex association problem between the unmanned vehicle and ground heterogeneous sensors on the basis of maximizing system throughput, and using precise potential energy game to prove the existence of Nash equilibrium; Step S7, designing a ground heterogeneous sensor alliance formation algorithm, combining the exchange criterion and the insertion criterion, and outputting a stable alliance structure; Step S8, converting the unmanned vehicle trajectory optimization model established in step S5 into a partially observable Markov decision process, combining the alliance structure output in step S7 to design a multi-agent collaborative learning trajectory optimization method to obtain the optimal unmanned vehicle path planning.

2. The multi-agent collaborative data collection and learning method based on alliance formation game according to claim 1 is characterized in that: Step S1 further comprises: Set up a simulation scenario, which includes a data collection center, N unmanned vehicles, M ground heterogeneous sensors and multiple obstacles The set of unmanned vehicles is represented as N = {1, 2..., N}, the set of ground heterogeneous sensors is represented as M = {1, 2, .., M}, and the available channels of unmanned vehicles are represented as Each unmanned vehicle independently selects a ground heterogeneous sensor alliance and channel to collect ground heterogeneous sensor data. D offloading data to a data collection center; The duration of the unmanned vehicle data collection is discretized into T equal time slots Δτ, and the position coordinates of the unmanned vehicle n in the tth time slot are It is expressed as: Where t = {1, 2, ..., T}, v represents the moving speed of the unmanned vehicle, d represents the displacement per unit time, represents the deflection angle of the unmanned vehicle n at the tth time slot; The following conditions are met: Where, d ugv,min represents the minimum collision distance between unmanned vehicles, d ob,min represents the minimum collision distance between the unmanned vehicle and the obstacle. and are the unmanned vehicle n′ and the obstacle Position coordinates at the tth time slot.

3. The multi-agent collaborative data collection and learning method based on alliance formation game according to claim 1 is characterized in that: Step S2 further comprises: Considering the impact of obstacles on the communication link, the channel gain between the unmanned vehicle n and the ground heterogeneous sensor m is expressed as: in, represents the distance between the unmanned vehicle n and the ground heterogeneous sensor m in the tth time slot, α is the path loss constant, β is the average channel gain; η represents the shadow effect and follows a Gaussian distribution The subscripts Los and NLos denote line-of-sight and non-line-of-sight, respectively; Assume that the communication between the unmanned vehicle and the ground sensor adopts the time division multiple access mode. In each time slot, the unmanned vehicle can only collect data from one ground heterogeneous sensor. The signal received by the unmanned vehicle n from the ground sensor m in the tth time slot is: Among them, P m represents the transmission power of the ground heterogeneous sensor m, represents the interference signal within the alliance, σ 2 is the white Gaussian noise power that affects the communication link between the unmanned vehicle and the sensor; the data transmission rate from the ground heterogeneous sensor m to the unmanned vehicle n in the tth time slot is: c n It represents the bandwidth of the data collected by drone n; The data unloading rate from the unmanned vehicle to the data collection center is defined as: in, Indicates that other unmanned vehicles n' are also on channel B D The interference caused by transmitting data on the n represents the transmission power of unmanned vehicle n, represents the white noise power when the unmanned vehicle unloads data, represents the channel gain from unmanned vehicle n to the data collection center.

4. The multi-agent collaborative data collection and learning method based on alliance formation game according to claim 1 is characterized in that: Step S3 further comprises: Constructing a dynamic model of the data of ground heterogeneous sensor m at time slot t+1 for: Among them, v m is the speed at which the ground heterogeneous sensor m collects data from the environment, is the amount of sensing data collected by the ground heterogeneous sensor m by the unmanned vehicle n in time slot t, is the association state between the unmanned vehicle n and the ground heterogeneous sensor m at time t, D m,max represents the data storage capacity of the ground heterogeneous sensor m; Construct a dynamic data model of the unmanned vehicle n at time slot t+1 for: in, represents the amount of data unloaded by unmanned vehicle n to the data collection center at time t, D n,max Represents the data storage capacity of unmanned vehicle n.

5. The multi-agent collaborative data collection and learning method based on alliance formation game according to claim 1 is characterized in that: In step S4, the Jain fairness index is introduced as an indicator to measure the fairness of ground heterogeneous sensor data transmission: in, Indicates relative collection amount; represents the amount of data sent by the ground heterogeneous sensor m to the unmanned vehicle n during a long-term mission; It represents the total amount of data collected by the ground heterogeneous sensor m from the environment before the current time slot t.

6. The multi-agent collaborative data collection and learning method based on alliance formation game according to claim 1 is characterized in that: In step S5, the unmanned vehicle trajectory optimization model in the data collection scenario is: In the formula, is the amount of sensing data collected by the ground heterogeneous sensor m by the unmanned vehicle n in time slot t, is the association state between the unmanned vehicle n and the ground heterogeneous sensor m at time t, l t is the position of the unmanned vehicle at time t, d ugv,min represents the minimum collision distance between unmanned vehicles, is the position coordinate of the unmanned vehicle n′ at the tth time slot, f t As an indicator to measure the fairness of ground heterogeneous sensor data transmission, represents the remaining power of the unmanned vehicle n in time slot t, and B represents the location set of obstacles.

7. The multi-agent collaborative data collection and learning method based on alliance formation game according to claim 1 is characterized in that: Step S6 further comprises: S61, let the ground heterogeneous sensors be participants and regard the unmanned vehicle-heterogeneous sensor association problem as a non-cooperative game Where M represents the set of participants, {S m } m∈M represents the strategy set of the participants, {u m } m∈M Denotes the utility function of the participant, and defines the utility function of the ground heterogeneous sensor m as the relative amount of data collection: in, represents the data transmission rate between the ground heterogeneous sensor m and the unmanned vehicle n at time slot t, represents the total amount of data collected from the environment by the ground heterogeneous sensor m to the current time slot t; then the alliance C i The profit function is: S62, consider maximizing the benefits of the alliance to design the alliance exchange criteria and joining criteria, where: The exchange criterion is: if the ground heterogeneous sensor s1∈C i , s2∈C j , when the ground heterogeneous sensors s1, s2 meet the following conditions, the alliance exchange operation is performed: The joining criterion is: if the ground heterogeneous sensor s1∈C i , when the ground heterogeneous sensor s1 meets the following conditions, it leaves the alliance C i Join Alliance C j : Among them, δ m represents the strategy of sensor m, δ -m represents the sensor association strategy except sensor m; S63, using exact potential energy game to prove the existence of Nash equilibrium.

8. The multi-agent collaborative data collection and learning method based on alliance formation game according to claim 1 is characterized in that: Step S7 further comprises: S71, randomly generate a set of unmanned vehicle position coordinates The coordinate set of ground heterogeneous sensors l m And the obstacle set Initialize the ground heterogeneous sensors and unmanned vehicle transmission power P m ,P n and bandwidth aggregation S72, calculating the distance from each ground heterogeneous sensor to each unmanned vehicle, and randomly selecting unmanned vehicles to form an initial alliance; S73, randomly select a ground heterogeneous sensor s1 and obtain the alliance C where the ground heterogeneous sensor s1 currently belongs i ; S74, randomly select an alliance C j ,C i ≠C j , in C j A ground heterogeneous sensor s2 is randomly selected; S75, Computing Alliance C i and C j The utility function and S76, determine whether the alliance update criteria are met, and if the alliance exchange criteria are met, update the alliance C i ←{C i / s1}∪s2; C j ←{C j / s2}∪s1; if the alliance joining criteria are met, update alliance C j ←C j ∪s1;C i ←C i / s1; S77, repeating steps S73 to S76, when the number of repetitions reaches the maximum number of iterations, outputting a stable alliance structure {C i } i∈N .

9. The multi-agent collaborative data collection and learning method based on alliance formation game according to claim 1 is characterized in that: Step S8 further comprises: S81, the unmanned vehicle trajectory optimization model is standardized as a partially observable Markov decision process, and the partially observable Markov decision process is represented as a tuple (o t ,s t ,a t ,o t+1 ,s t+1 ,r t ), where o represents the observation space, s represents the global state space, a represents the action space, and the reward function r represents the evaluation of the current observation action pair (o, a); S82, combined with the alliance structure, designs a trajectory optimization method based on multi-agent collaborative learning. The design process includes the following steps: S821, determine the model parameters of the optimizer Adam, which include learning rate α, discount factor γ, strategy clipping range ε ​​and experience pool Initialize the Actor network parameters π of each unmanned vehicle n,θ and Critic network parameters V n,θ ; S822, initialize the observation space of each unmanned vehicle S823, obtain the association between the unmanned vehicle and heterogeneous sensors S824, using the Actor strategy network based on the current observation results Output probability distribution Sampling actions from probability distributions and the corresponding probability Set the selected action Input into the environment to obtain the observation space for the next time slot And calculate the reward of the current state based on the feedback from the environment S825, use the Critic network to calculate the value of the current action S826, repeat steps S822 to S825 to obtain the current observation, global state, action, action probability, reward, next observation and global state of each unmanned vehicle (o t ,s t ,a t ,π t ,r t ,o t+1 ,s t+1 ) ; Store some observable Markov decision processes into the experience pool S827, update model parameters, process batch data, calculate advantage function, perform strategy pruning, and maintain the stability of the update strategy; S828, repeating steps S823 to S827 until all unmanned vehicles run out of power or the mission is completed; S829: When the maximum number of training times is reached, the optimal unmanned vehicle trajectory is output

Images