AoI-aware incentive method for privacy-constrained railway data sharing

Abstract

The application discloses an AoI-aware incentive method for railway data sharing under privacy constraints, and comprises the following steps: a railway data sharing model is constructed, including a data requester and multiple data providers; a utility maximization problem is constructed with the aim of maximizing the self-utility of the data requester and the data providers, including an incentive payment strategy and a data uploading strategy; the utility maximization problem is modeled as a multi-agent Markov decision process; the optimal strategy of each agent is learned by using IMASAC, the multi-agent Markov decision process is solved, and the optimal incentive payment of the data requester and the optimal data uploading amount of the data providers are obtained. The application can depict freshness-aware double-layer requester-provider interaction and avoid the privacy bottleneck caused by centralized critics.

Description

Technical Field

[0001] This invention relates to the field of railway information sharing technology, and in particular to an AoI perception incentive method for railway data sharing under privacy constraints. Background Technology

[0002] With the rapid development of railway information systems, complex intelligent railway operation tasks such as multimodal heterogeneous data fusion and dynamic train scheduling in intelligent tunnel maintenance increasingly rely on heterogeneous operational data generated by multiple physically isolated departments and business processes. This data supports critical operational decisions such as train scheduling adjustments, passenger flow organization, and fault response. In these scenarios, data availability is not the only factor; data timeliness is equally crucial. Outdated scheduling data, passenger flow data, or equipment status data can weaken real-time train adjustment, station organization, and maintenance scheduling capabilities. Therefore, railway data sharing is not merely a data circulation issue, but a key support for achieving timely and collaborative railway operation decisions. However, actual railway data sharing typically involves multiple stakeholders, including data requesters, data providers, and sometimes third-party intermediary platforms. Different departments (such as engineering and maintenance departments, meteorological departments, and signaling centers) possess their own operational data and participate in sharing based on highly heterogeneous costs, responsibilities, and privacy concerns. Since data collection, storage, preprocessing, and uploading all incur costs and may increase the risk of information exposure, a reasonable incentive mechanism is needed to compensate data providers, improve their utility, and maintain their continued enthusiasm for participating in railway data sharing.

[0003] In real-time intelligent railway operations, cross-departmental data sharing is a crucial link supporting timely decision-making, but strict data confidentiality requirements severely limit data flow. Information age (AoI) is critical for such safety-critical applications, but existing incentive mechanisms largely ignore the dynamic decay of data value over time.

[0004] Furthermore, mainstream multi-agent reinforcement learning (MARL) methods typically employ a centralized training and distributed execution architecture. Such architectures may expose joint actions or global states in physically isolated railway subsystems, thereby posing an unacceptable risk of privacy breaches.

[0005] Most existing incentive mechanisms primarily focus on the relationship between the amount of data uploaded and incentive payments; that is, requesters pay rewards to providers to incentivize them to contribute more data. However, in railway operation decision-making, more data does not necessarily lead to better decisions if the data is outdated. For example, real-time scheduling, passenger flow control, and safety response all require fresh information within a limited decision window. Existing research has shown that AoI (Aspect of Information) is a key indicator in real-time systems because data value depends on timeliness. Although the importance of AoI has been recognized, existing incentive designs rarely consider freshness as a first-order strategic decision factor. Therefore, AoI should be explicitly incorporated into incentive mechanisms, rather than merely as an external performance indicator. More importantly, existing AoI-related research has not addressed the requester-provider hierarchical decision-making structure, nor has it considered the problem of private information asymmetry among participants, thus making it difficult to directly support privacy-constrained railway data sharing with two-tiered strategic interactions.

[0006] Most existing learning incentive mechanisms employ a centralized training and distributed execution (CTDE) architecture or do not utilize a fully independent experience replay buffer. While CTDE is effective in many multi-agent learning tasks, it may conflict with the privacy requirements of railway data sharing. Centralized commentators typically need access to joint actions, global states, or cross-agent trajectories during training; this information aggregation may leak or allow inferences about the provider's cost structure, behavioral patterns, and utility-related parameters. Furthermore, while federated learning helps avoid raw data leakage, it incurs high global model aggregation overhead when coordinating continuous, high-frequency game-like incentive allocations between two layers of decision-makers and struggles to handle non-stationary multi-agent exploration. Therefore, mainstream CTDE paradigms and traditional federated aggregation create privacy and efficiency bottlenecks in privacy-constrained railway operations. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, this invention provides an AoI-aware incentive method for privacy-constrained railway data sharing, which solves the problems of existing incentive mechanisms ignoring the dynamic decay of data value over time and potentially leading to privacy leakage risks.

[0008] To achieve the aforementioned objectives, the technical solution adopted by this invention is: an AoI (Awareness and Incentive) method for privacy-constrained railway data sharing, comprising: Construct a railway data sharing model, including one data requester and multiple data providers; With the goal of maximizing the utility of both data requesters and data providers, a utility maximization problem is constructed, including incentive payment strategies and data upload strategies. The utility maximization problem is modeled as a multi-agent Markov decision process. By using IMASAC to learn the optimal policy of each agent, we solve the multi-agent Markov decision process to obtain the optimal incentive payment for the data requester and the optimal amount of data to be uploaded for the data provider.

[0009] The beneficial effects of this invention include: 1. By reformulating the utility maximization problem as a multi-agent (MDP) problem and proposing IMASAC within a DTDE architecture with independent experience replay buffers, this invention enables agents to learn equilibrium-consistent policies from local interactions, circumventing the difficulty of obtaining accurate and complete environmental information required by existing technologies. Extensive simulation experiments demonstrate that the proposed method achieves stable convergence, close to SE performance, and consistently outperforms existing technologies in both requester and provider utilities, providing a feasible incentive learning scheme for practical railway systems.

[0010] 2. By proposing an independent multi-agent soft actor-critic algorithm, which strictly adopts a distributed training and distributed execution (DTDE) architecture and is equipped with an independent experience replay buffer, it can mitigate the risk of privacy leakage and improve scalability under incomplete information without the need for a centralized critic or global state aggregation, providing a promising approach for the design of incentive mechanisms under incomplete information. Attached Figure Description

[0011] Figure 1 A flowchart of an AoI perception incentive method for privacy-constrained railway data sharing is provided for an embodiment. Figure 2 This diagram illustrates the convergence performance of IMASAC under different learning rates. Figure 3 This is a schematic diagram illustrating the convergence performance of IMASAC under different historical lengths. Figure 4 The SE and convergence results of the algorithm proposed in this invention are shown. Figure 5 This is a utility trajectory diagram of the data requester and the data provider during the training process; Figure 6 Provides a comparison chart of side-by-side and request-side utility for different MARL baselines and architectures. Detailed Implementation

[0012] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0013] like Figure 1 As shown, in one embodiment of the present invention, an AoI-aware incentive method for privacy-constrained railway data sharing includes the following steps: S1. Construct a railway data sharing model, including one data requester and multiple data providers.

[0014] This invention considers a railway data sharing framework consisting of one task initiator (i.e., the data requester) and N responders (i.e., the data providers). The data requester publishes a data request task, and the data providers upload the required data to complete the task.

[0015] S2. With the goal of maximizing the utility of both the data requester and the data provider, construct a utility maximization problem, including incentive payment strategies and data upload strategies.

[0016] In the data sharing process, both the data requester and the data provider aim to maximize their own utility. This invention models the interaction as a two-step decision-making process: (a) The task initiator determines the unit incentive payment, that is, the reward paid to the unit upload strategy of the provider; (b) Each provider selects an upload strategy consisting of the amount of data uploaded and the age of the information.

[0017] Each provider has a utility function that characterizes the benefits it gains from participating in the data-sharing task. The utility of provider n is defined as the sum of its intrinsic utility and the incentive payments it receives from the requesters, minus the costs it incurs.

[0018] =

[0019]

[0020]

[0021] in, Indicates intrinsic utility; Used to characterize the inherent benefits of data uploading; This is a time-dependent decay factor. This is a task-related decay coefficient, an exponential form that naturally characterizes the decay of information value with age while maintaining parsing processability. Therefore, provider utility decreases as AoI increases, thus encouraging the uploading of fresher data. This is the cost function.

[0022] In summary, the provider's utility function can be expressed as: .

[0023] Similarly, the utility of the requester is defined as the benefit derived from receiving the amount of uploaded data minus the total incentives paid to all providers: ; The first term represents the revenue generated from data collection, and the second term represents the total incentive paid to the data provider. The first term increases as the amount of data received increases. >0 is a system adjustment parameter, specifically a scaling factor for the requester's revenue item. This is achieved through adjustment... It can control the trade-off between the requester's benefits and incentive payments, and ensure that the requester's utility is non-negative.

[0024] In the railway data sharing scenario, data requesters and data providers maximize their respective utility by negotiating incentive payment strategies and data upload strategies.

[0025] The requester's target incentive payment strategy P1 is as follows:

[0026] In the formula, The objective function for incentivizing payment strategies; For the utility function of the data requester; Indicates constraints; For data providers The amount of data uploaded; For data providers Corresponding information age; For the data requester to the data provider Incentive payments to employees; A collection of data providers ; A collection of incentive payments for data providers. ; The collection of data uploads from data providers. ; The data provider's information age set, ; Adjust system parameters; Incentive payments to data requesters; The unit incentive coefficient is related to the information age. ; The unit incentive coefficient is related to the amount of uploaded data. .

[0027] The data provider's target data upload strategy P2 is as follows:

[0028] in,

[0029] In the formula, Let the objective function be the data upload strategy; A utility function for the data provider; , These are all parameters related to data quality. , ; The unit cost coefficient related to the age of the information. ; This is a unit cost coefficient related to the amount of data uploaded. ; For data providers The minimum feasible information: age; For data providers The maximum feasible information is age.

[0030] S3. Model the utility maximization problem as a multi-agent Markov decision process.

[0031] Specifically, it includes: The data requester and each data provider are modeled as independent SAC agents; the states, actions, and immediate rewards of the data requester agent and the data provider agent are defined; the optimization objectives of the data requester agent and the data provider agent are defined; the optimization objective of the data requester agent is the incentive payment strategy, and the optimization objective of the data provider agent is the data upload strategy.

[0032] Each independent SAC agent includes: An actor network is used to output a stochastic strategy for continuous actions. The critic network is used to mitigate overestimation bias; the critic network employs a dual-Q network.

[0033] Entropy temperature is used to balance between exploitation and exploration and to adaptively update temperature parameters.

[0034] The state, actions, and immediate rewards of the data requesting agent are as follows:

[0035]

[0036]

[0037] In the formula, In time step The state of the data requesting agent; In time step The incentive payment is determined by the data requester based on its own historical incentive payments and the historical upload volume and information age strategy of all data providers. ; In time step The information age strategy is determined by the data provider based on the data requester's historical incentive payments and the historical data upload volume and information age strategy of all data providers; In time step The data provider determines the amount of data to be uploaded based on the data requester's historical incentive payments, the historical upload data volume of all data providers, and the information age strategy. The historical incentive payment sequence for the data requester; The historical upload volume of the data provider; A historical information age strategy for data providers; For time step The actions of the data requesting agent; For time step Instant rewards for the data requesting agent.

[0038] The state, actions, and immediate rewards of the data provider agent are as follows:

[0039]

[0040]

[0041] In the formula, In time step Below, the data provider The state; In time step Below, the data provider The amount of data uploaded by the data requester is determined based on the historical incentive payments of the data requester and the historical upload data volume of other data providers, as well as the information age strategy. In time step Data Provider The information age strategy is determined based on the historical incentive payments of the data requester and the historical data upload volume and information age strategy of other data providers. For data providers In addition, the historical upload volume of other data providers; For data providers In addition, other data providers' historical information age strategies; In time step Below, the data provider The action; In time step Below, the data provider Instant rewards; In time step The data provider is determined by the data requester based on its historical incentive payments and the historical upload volume and information age strategy of other data providers. Incentive payments.

[0042] It should be noted that the historical policies contained in the state are treated as publicly observable interaction records, rather than private utility parameters. Therefore, the learning process does not need to disclose the private utility functions of the participants.

[0043] S4. Use IMASAC to learn the optimal policy of each agent, solve the multi-agent Markov decision process, and obtain the optimal incentive payment for the data requester and the optimal amount of data to be uploaded for the data provider.

[0044] Specifically, the steps include: A1. At each time step, the data requesting agent is based on its state. Sampling payment actions from random strategies output by the actor network. This involves incentivizing payment strategies and broadcasting observable payment information. A2, based on the state of each data provider agent. Sample actions from a random strategy output by the actor network. That is, the combined strategy of uploaded data volume and AoI. ; A3. Calculate the immediate rewards for the data requesting agent and the data providing agent, and then proceed to the next state; A4. Each agent stores its own transferred samples into the corresponding private experience replay buffer; IMASAC uses a DTDE architecture, which configures an independent experience replay buffer for each agent. In this system, each data requesting agent maintains a private experience buffer; each data provider agent maintains a private experience buffer; each private experience buffer stores only local transfer tuples, and its expression is:

[0045] In the formula, For time step Below, intelligent agents The state; For time step Below, intelligent agents The action; For time step Below, intelligent agents Instant rewards; For time step Below, intelligent agents The state.

[0046] Unlike CTDE-type MARL methods that rely on centralized training signals, our DTDE architecture trains a local actor-critic pair for each participant and uses an independent experience replay buffer to mitigate privacy risks. Specifically, each agent stores its own interaction samples only in a private buffer and independently updates the actor-critic parameters based on local transfer samples, without exchanging experience data with other participants. Therefore, IMASAC minimizes cross-agent information sharing during training, is suitable for privacy-constrained railway data sharing, and maintains good scalability as the number of participants increases.

[0047] A5. When the number of samples accumulated in each private experience replay buffer exceeds the preset threshold, each agent performs random batch sampling from its maintained private experience replay buffer and updates the parameters of IMASAC. Repeat steps A1-A5 until the random policy output by the actor network converges, thus obtaining the optimal incentive payment for the data requester and the optimal amount of data to be uploaded for the data provider.

[0048] Update the parameters of IMASAC, including: Commentator Update: Sample the next action in the next state; The target value of the next state is calculated based on the next action, and its expression is:

[0049] In the formula, The target value for the next state; For immediate rewards in locally transferred tuples; Discount factor; For intelligent agents The One target commentator; The next state; The action for the next state; This indicates that the smaller value among the two commentators is taken; It is the entropy temperature coefficient; Indicates the target actor's network output action. The logarithmic probability of , where the base of the log function is the natural constant e; The loss function of the current commentator network is constructed based on the target value of the next state, and its expression is as follows:

[0050] In the formula, Let be the loss function of the current critic network; This indicates a demand for expectation; Indicates the current Q network; This is the current state; For the current action; Based on the commentator network loss function, the parameters of the commentator network are updated using gradient descent until the loss function converges. Actor Update: The actor network loss function is constructed as follows:

[0051] In the formula, For the actor network loss function; Based on the actor network loss function, the actor network parameters are updated using gradient descent until the loss function converges. Temperature update: Define the target entropy; The temperature loss function is constructed based on the target entropy, and its expression is as follows:

[0052] In the formula, Represents the temperature loss function; Represents the target entropy; Target network soft update: After updating the parameters of the critic network and the actor network, perform a parameter adjustment once for the target critic network and the target actor network for each agent, with the expression as follows:

[0053] In the formula, These are the network parameters after parameter adjustment. This is the soft update coefficient. ; For target network parameters; These are the current network parameters.

[0054] To verify the effectiveness of this invention, numerical simulations were performed in a railway data sharing scenario. This scenario was modeled as a single-leader, multi-follower Stackelberg game involving one data requester and five heterogeneous data providers. Each training round contained 50 time steps, and the number of training rounds was set to 500. The environmental parameters were set as follows: discount factor... =0.95, target network soft update coefficient τ = 0.003, AoI decay coefficient = 0.1, incentive correlation coefficient = 0.1 and = 1.0, Requester's revenue scaling factor =2.5. The AoI range is limited to [t]. min , t max = [0.5, 7.0]. To reflect the heterogeneity of data providers, the intrinsic utility parameter and cost coefficient are set as a = [1.2, 1.3, 1.0, 1.1, 0.8] and b = [4.5, 3.0, 7.0, 5.5, 4.0]; =[1.0, 0.9, 0.8, 0.6, 0.7] = [2.3, 1.3, 2.8, 2.0, 2.5]. These heterogeneous parameters are used to simulate real-world differences in data quality and upload costs within the railway sector.

[0055] In terms of algorithm structure, each agent adopts the SAC framework, including a random actor network and two commentator networks, with a corresponding target network for each commentator. All networks are fully connected, containing two hidden layers with 128 neurons each, and using ReLU activation. Regarding training hyperparameters, the learning rate for both actors and commentators is set to 3×10⁻⁶. 4 The entropy temperature learning rate is set to 1×10. 4 The experience replay buffer capacity is 10. 6 The mini-batch size is 128. To meet privacy and scalability requirements, all agents run on the DTDE architecture and maintain independent experience replay buffers. Entropy temperature is automatically adjusted during training.

[0056] The IMASAC provided by this invention is compared with the following baselines and analytical SE benchmarks: MASAC-CTDE, i.e., the SAC baseline under the CTDE architecture, used to compare CTDE and DTDE in the same soft actor-commentator framework; MADDPG-CTDE and MADDPG-DTDE, used to evaluate the impact of the architecture on deterministic policy learning; MATD3-CTDE and MATD3-DTDE, as a stronger TD3-type deterministic MARL comparison; and the SE benchmark, i.e., analytical Stackelberg equilibrium under complete information, used only as an oracle reference for post-training evaluation.

[0057] Figure 2 The convergence behavior of IMASAC under different learning rates δ is demonstrated. Figure 2As shown in (a), the total utility of the data provider increases rapidly in the early stages under all learning rate settings, indicating that the algorithm can effectively explore the policy space and quickly improve performance. However, there are significant differences in convergence speed and stability among different settings. When δ = 3 × 10 3 When δ = 3×10, the algorithm converges faster, but early training oscillations are significant; 5 At this rate, training is smoother but converges more slowly, requiring more rounds to reach a steady state. In contrast, a moderate learning rate δ = 3 × 10⁻⁶ is more effective. 4 A good balance is achieved between convergence speed and stability, resulting in more stable and better long-term performance. Figure 2 (b) The requester utility also exhibits a similar pattern. A larger learning rate leads to larger parameter update amplitudes, resulting in more severe fluctuations in the early stages; a smaller learning rate leads to slower convergence. Considering both convergence speed and training stability, subsequent experiments used δ = 3 × 10⁻⁶. 4 .

[0058] Figure 3 The convergence performance of IMASAC under different history lengths L is demonstrated. Figure 3 As shown in (a), the total utility of the data provider changes significantly with L. When L = 2, convergence is slow and the final steady-state utility is low; increasing the history length to L = 3 significantly improves both convergence speed and final utility. However, further increasing to L = 4 does not bring significant performance gains, but instead induces more pronounced fluctuations in the early stages. This indicates that historical information helps enhance state representation and improve decision quality, but excessively long histories increase state dimensionality and may compromise training stability. The utility of the requester is... Figure 3 A similar trend is observed in (b). When historical data is insufficient, the agent cannot fully capture the dynamic interaction structure in the Stackelberg game; when L = 3, the system reaches its highest stable utility, indicating that an adequate amount of historical data can effectively capture the strategic dependence between the leader and followers; when L = 4, the expansion of the state space leads to an increase in the variance of policy updates, and the performance improvement tends to saturate. Therefore, this invention selects L = 3 in subsequent experiments.

[0059] To illustrate how an agent gradually learns a stable policy, Figure 4 It demonstrates the evolution of the amount of data uploaded, AoI, and incentive payments during the training process. Figure 4 (a) Give the upload data volume x of all providers. n , Figure 4 (b) Give the corresponding AoI decision t n , Figure 4(c) Provide the incentive payments published by the requesting party to each provider. The dashed line represents the analytical SE derived theoretically, used to evaluate whether the learned strategy is close to the equilibrium strategy. It can be seen that the training is roughly divided into an early exploration phase and a later stabilization phase. In the early phase, the leader and followers explore the continuous action space with large parameter updates, resulting in significant oscillations in all three types of actions; some providers may temporarily choose larger upload volumes or higher AoIs, while the leader continuously adjusts the incentives based on the provider behavior. As training progresses, action oscillations gradually decrease, and all actions converge to a stable level close to the SE benchmark, indicating that IMASAC can learn an equilibrium consistent strategy through repeated interactions without accessing the private parameters of other participants.

[0060] To further verify the convergence performance, Figure 5 The utility trajectories of the requester and provider during the training process are presented. Figure 5 (a) Show the utility of each provider Figure 5 (b) Showing the requester's utility. The dashed line represents the equilibrium utility obtained from theoretical analysis. It can be seen that in the early stages, due to the instability of continuous exploration and action adjustment, the provider's utility fluctuates significantly, with some providers even experiencing negative utility. This is reasonable because the agent has not yet learned the optimal response structure during the exploration phase, and there may be a mismatch between uploaded actions and incentives. As training progresses, the oscillations gradually weaken, and the utility of all providers converges to a stable level close to the SE benchmark, generally reaching convergence after approximately 200 rounds, indicating good learning efficiency. The requester's utility also fluctuates significantly in the early stages, especially when the incentive payoff changes significantly, because a higher total payoff burden temporarily reduces its utility. As the provider's strategy stabilizes, the requester can adjust incentives more precisely, and its utility gradually increases and stabilizes near the theoretical equilibrium value.

[0061] These results demonstrate that IMASAC can effectively learn the optimal incentive mechanism under a two-level decision-making structure. The simultaneous convergence of utility for both parties further illustrates the dynamic coupling between incentive payments and upload behavior: increasing incentives can stimulate greater upload volume and lower AoI, thereby improving provider utility, but the increased payment may temporarily reduce leader utility; conversely, decreasing incentives reduces provider benefits and weakens their participation. Through repeated interactions, the agent learns to balance payment costs and upload benefits, enabling the system to reach a stable operating point where both parties' utilities are close to equilibrium.

[0062] To evaluate the advantages of IMASAC compared to existing methods, Figure 6 The results of its comparison with multiple MARL baselines and analytical SE benchmarks are summarized. Figure 6 (a) Report the total utility of all providers. Figure 6(b) Report requester utility, where the dashed line represents the SE baseline. Overall, IMASAC achieves the best performance, with its utility curve closest to the SE. From an algorithmic perspective, IMASAC exhibits higher stability and better asymptotic performance than deterministic actor-critic baselines such as MADDPG and MATD3. This can be attributed to the maximum entropy objective of SAC, which maintains sufficient stochastic exploration in the continuous and coupled Stackelberg decision space, thus mitigating the problem of premature convergence to suboptimal policies in early training. From an architectural perspective, under the same backbone algorithm, the DTDE architecture generally exhibits better stability and steady-state performance than the CTDE architecture: IMASAC outperforms MASAC-CTDE, and the DTDE-based variants of MADDPG and MATD3 also reduce oscillations and improve convergence compared to their CTDE counterparts. This demonstrates that, in the privacy-constrained heterogeneous multi-agent environment considered in this invention, centralized training utilizing joint / global information (such as centralized commentator or shared experience aggregation) may exacerbate non-stationarity and estimation errors, while the DTDE architecture with independent actor-commentator updates and a local experience replay buffer can deliver more robust learning dynamics. Overall, the combination of SAC, the DTDE architecture, and the independent experience replay buffer enables IMASAC to achieve the most robust convergence and closest to analytical SE among all comparison baselines.

[0063] This invention models the interaction between a data requester and multiple heterogeneous data providers as a two-stage single-leader-multiple-follower Stackelberg game, where the requester announces incentive decisions, and heterogeneous providers choose their data upload volume and AoI. To avoid reliance on private utility parameters of providers in practice, this invention reformulates the game as a multi-agent MDP and proposes IMASAC within a DTDE architecture with independent experience replay buffers, enabling agents to learn equilibrium-consistent policies from local interactions. Extensive simulation experiments demonstrate that the proposed method achieves stable convergence, close to SE performance, and consistently outperforms representative MARL baselines such as MADDPG and MATD3 in terms of both requester and provider utility.

Claims

1. A method for AoI (Aspect-Oriented Incentive) perception incentives for privacy-constrained railway data sharing, characterized in that, include: Construct a railway data sharing model, including one data requester and multiple data providers; With the goal of maximizing the utility of both data requesters and data providers, a utility maximization problem is constructed, including incentive payment strategies and data upload strategies. The utility maximization problem is modeled as a multi-agent Markov decision process. By using IMASAC to learn the optimal policy of each agent, we solve the multi-agent Markov decision process to obtain the optimal incentive payment for the data requester and the optimal amount of data to be uploaded for the data provider.

2. The method according to claim 1, characterized in that, The specific details of the incentive payment strategy P1 are as follows: in, In the formula, The objective function for incentivizing payment strategies; For the utility function of the data requester; Indicates constraints; For data providers The amount of data uploaded; For data providers Corresponding information age; For the data requester to the data provider Incentive payments to employees; A collection of data providers ; A collection of incentive payments for data providers. ; The collection of data uploads from data providers. ; The data provider's information age set, ; Adjust system parameters; Incentive payments to data requesters; The unit incentive coefficient is related to the information age. ; The unit incentive coefficient is related to the amount of uploaded data. .

3. The method according to claim 2, characterized in that, The data upload strategy P2 is as follows: in, In the formula, Let the objective function be the data upload strategy; A utility function for the data provider; , These are all parameters related to data quality. , ; The unit cost coefficient related to the age of the information. ; This is a unit cost coefficient related to the amount of data uploaded. ; For data providers The minimum feasible information: age; For data providers The maximum feasible information is age.

4. The method according to claim 3, characterized in that, The utility maximization problem is modeled as a multi-agent Markov decision process, including: Model the data requester and each data provider as an independent SAC agent; Define the state, actions, and immediate rewards of the data requesting agent and the data providing agent; Define the optimization objectives of the data requester agent and the data provider agent; the optimization objective of the data requester agent is the incentive payment strategy, and the optimization objective of the data provider agent is the data upload strategy.

5. The method according to claim 4, characterized in that, The state, actions, and immediate rewards of the data requesting agent are as follows: In the formula, In time step The state of the data requesting agent; In time step The incentive payment is determined by the data requester based on its own historical incentive payments and the historical upload volume and information age strategy of all data providers. ; In time step The information age strategy is determined by the data provider based on the data requester's historical incentive payments and the historical data upload volume and information age strategy of all data providers; In time step The data provider determines the amount of data to be uploaded based on the data requester's historical incentive payments, the historical upload data volume of all data providers, and the information age strategy. The historical incentive payment sequence for the data requester; The historical upload volume of the data provider; A historical information age strategy for data providers; For time step The actions of the data requesting agent; For time step Instant rewards for the data requesting agent.

6. The method according to claim 5, characterized in that, The state, actions, and immediate rewards of the data provider agent are as follows: In the formula, In time step Below, the data provider The state; In time step Below, the data provider The amount of data uploaded by the data requester is determined based on the historical incentive payments of the data requester and the historical upload data volume of other data providers, as well as the information age strategy. In time step Data Provider The information age strategy is determined based on the historical incentive payments of the data requester and the historical data upload volume and information age strategy of other data providers. For data providers In addition, the historical upload volume of other data providers; For data providers In addition, other data providers' historical information age strategies; In time step Below, the data provider The action; In time step Below, the data provider Instant rewards; In time step The data provider is determined by the data requester based on its historical incentive payments and the historical upload volume and information age strategy of other data providers. Incentive payments.

7. The method according to claim 4, characterized in that, Each independent SAC agent includes: An actor network is used to output a stochastic strategy for continuous actions. The critic network is used to mitigate overestimation bias; the critic network employs a dual-Q network. Entropy temperature is used to balance between exploitation and exploration and to adaptively update temperature parameters.

8. The method according to claim 7, characterized in that, IMASAC uses a DTDE architecture, which configures an independent experience replay buffer for each agent. In this system, each data requesting agent maintains a private experience buffer; each data provider agent maintains a private experience buffer; each private experience buffer stores only local transfer tuples, and its expression is: In the formula, For time step Below, intelligent agents The state; For time step Below, intelligent agents The action; For time step Below, intelligent agents Instant rewards; For time step Below, intelligent agents The state.

9. The method according to claim 8, characterized in that, Using IMASAC to learn the optimal policy for each agent, we solve the multi-agent Markov decision process to obtain the optimal incentive payment for the data requester and the optimal amount of data to be uploaded for the data provider. This includes the following steps: A1. At each time step, the data requesting agent is based on its state. Sampling payment actions from random strategies output by the actor network. This involves incentivizing payment strategies and broadcasting observable payment information. A2, based on the state of each data provider agent. Sample actions from a random strategy output by the actor network. That is, the combined strategy of uploaded data volume and AoI. ; A3. Calculate the immediate rewards for the data requesting agent and the data providing agent, and then proceed to the next state; A4. Each agent stores its own transferred samples into the corresponding private experience replay buffer; A5. When the number of samples accumulated in each private experience replay buffer exceeds the preset threshold, each agent performs random batch sampling from its maintained private experience replay buffer and updates the parameters of IMASAC. Repeat steps A1-A5 until the random policy output by the actor network converges, thus obtaining the optimal incentive payment for the data requester and the optimal amount of data to be uploaded for the data provider.

10. The method according to claim 9, characterized in that, Update the parameters of IMASAC, including: Commentator Update: Sample the next action in the next state; The target value of the next state is calculated based on the next action, and its expression is: In the formula, The target value for the next state; For immediate rewards in locally transferred tuples; Discount factor; For intelligent agents The One target commentator; The next state; The action for the next state; This indicates that the smaller value among the two commentators is taken; It is the entropy temperature coefficient; Indicates the target actor's network output action. The logarithmic probability of , where the base of the log function is the natural constant e; The loss function of the current commentator network is constructed based on the target value of the next state, and its expression is as follows: In the formula, Let be the loss function of the current critic network; This indicates a demand for expectation; Indicates the current Q network; This is the current state; For the current action; Based on the commentator network loss function, the parameters of the commentator network are updated using gradient descent until the loss function converges. Actor Update: The actor network loss function is constructed as follows: In the formula, For the actor network loss function; Based on the actor network loss function, the actor network parameters are updated using gradient descent until the loss function converges. Temperature update: Define the target entropy; The temperature loss function is constructed based on the target entropy, and its expression is as follows: In the formula, Represents the temperature loss function; Represents the target entropy; Target network soft update: After updating the parameters of the critic network and the actor network, perform a parameter adjustment once for the target critic network and the target actor network for each agent, with the expression as follows: In the formula, These are the network parameters after parameter adjustment. This is the soft update coefficient. ; For target network parameters; These are the current network parameters.

Images