A method and system for joint simulation of intelligent game and communication network

Abstract

A kind of joint simulation method of intelligent game and communication network, nodes are separately deployed using virtualization and container technology, so that each node has independent and scalable wireless communication module, shared memory module, intelligent engine module and time synchronization module, 1) construct the game model of wireless communication;2) initialize the joint simulation system of intelligent game and communication network;3) run the joint simulation system of intelligent game and communication network, nodes start to communicate transmission and perceive surrounding communication state information, such as routing state, channel state, throughput, packet loss rate etc.;4) each node uses shared memory module to pass the perceived communication state information to intelligent engine, to realize fast data transmission;5) intelligent engine module on each node starts to work, after receiving the communication state information perceived by node, according to existing model or rule to reason and decision.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of wireless communication network simulation, and particularly relates to a joint simulation method of intelligent game and communication network. BACKGROUND

[0002] The joint simulation of intelligent game and communication network is a comprehensive simulation method that combines intelligent decision-making process and communication network characteristics, aiming to evaluate and optimize the effect of game strategies in actual communication network environment. Specifically, each communication node acts as a game player, and through strategy selection, the resource utilization rate, throughput and other communication indicators are improved under the condition of limited communication resources. During the game process, each communication node adjusts the strategy based on its own observation information, so that the node finds a balance between local optimization and global optimization.

[0003] Common communication network simulation tools include OMNet++, OPNET and NS-3, which can simulate different types of networks and real communication processes; the research of intelligent game mainly focuses on combining traditional game theory with intelligent algorithms such as machine learning and reinforcement learning, to solve resource competition and strategy optimization problems. With the development of 5G, 6G network and Internet of Vehicles, the demand for research on cooperation and confrontation in multi-user and complex environment has increased significantly, and the existing joint simulation of intelligent game and communication network still has the following problems in practical application: 1) The simulation verification of intelligent game and communication network joint mainly focuses on the behavior level, and lacks joint simulation of behavior level and signal level, so the fidelity of the method cannot be verified; 2) For large-scale joint simulation of intelligent game and communication network, a large number of communication nodes and complex communication environment need to be simulated, and the performance overhead of processing node information and making decisions will be huge, while many researches do not consider the system overhead when performing signal level simulation in large-scale scenarios, resulting in limited simulation scale and insufficient scalability; 3) In the multi-agent scenario and dynamic communication environment, the learning and decision-making of intelligent game is difficult to generate in real time, and most current researches do not consider the limitation of computing power, ignoring the real-time performance of the algorithm, so it is difficult to apply to wireless communication networks with dynamic environment.

[0004] CN201810458873.5 discloses a communication network abnormal user detection method based on a generative adversarial network. First, the data of the abnormal user is regularized to obtain data with consistent dimensions and magnitudes. The generative adversarial network is trained, i.e. the generator and discriminator are trained to realize oversampling of the abnormal user. The data generated by the generator and the normal user form a training data set, and a deep fully connected neural network is used to classify the training data set to determine the user type. Through the mutual game training mode between the neural networks in the generative adversarial network, the distribution of the abnormal user data is approximated to realize abnormal user detection and solve the technical problem of the imbalance of the proportion of the training data set affecting the training effect of the classification model. SUMMARY

[0005] The purpose of the present application is to overcome the problems of system resource limitation, poor scalability and poor real-time performance of existing intelligent game and communication network joint simulation methods in the face of large-scale and dynamically changing communication environments. The present application proposes an intelligent game and communication network joint simulation method and system, which combines communication and intelligence. Based on the communication network module, the nodes are separated from the communication network simulation system by means of virtualization technology to construct an intelligent engine module, so that each node can independently make autonomous intelligent decisions, improving the scalability of the system. A shared memory module and a time synchronization module are constructed to realize fast interaction between the wireless communication module and the intelligent engine module, and the impact of the computing power limitation and the time overhead is calculated in the actual communication system to improve the operation efficiency of the method and the fidelity of the system.

[0006] To achieve the above-mentioned purpose, the technical solution proposed by the present application is a joint simulation method of intelligent game and communication network. For each node in the communication network, the node is deployed separately using virtualization and container technology, so that each node has mutually independent and scalable wireless communication modules, shared memory modules, intelligent engine modules and time synchronization modules, and shares the same wireless communication environment. The scene of the joint simulation system of intelligent game and communication network is as shown in Figure 1 , and the structure of the intelligent communication node is as shown in Figure 2 . The overall process of the scheme is as follows:

[0007] Step 1: Construct a wireless communication game model, the specific steps of which are as follows:

[0008] 1) Determine the participants Γ={1,2,…,n} in the wireless communication game model, and the participants are nodes in the communication network;

[0009] 2) Define the strategy set S i of each participant, such as power control, spectrum selection, resource allocation, etc.

[0010] 3) Define the utility function U of each player i such as transmission rate, signal to interference ratio, resource utilization, etc.

[0011] 4) Select the type of game, such as static game, dynamic game, incomplete information game, etc.

[0012] where n represents the number of participants, and subscript i represents any participant.

[0013] Step two: Initialize the joint simulation system of intelligent game and communication network, as shown in Figure 1 At the system level, initialize the total time length of the simulation, the number of nodes, and the parameters such as the geographical location, movement model and node type of each node; at the communication level, initialize the traffic model of the application layer, the TCP or UDP protocol of the transmission layer, the routing protocol and topology structure of the network layer, the MAC protocol of the MAC layer and the channel, power and other parameters of the physical layer; at the intelligent level, initialize the knowledge base, reasoning engine and learning engine. According to the above initialization, the wireless communication module will establish the corresponding communication link for each node according to the geographical location and network topology of the node.

[0014] Step three: Run the joint simulation system of intelligent game and communication network, and the nodes start communication transmission and perceive the surrounding communication state information such as routing state, channel state, throughput, packet loss rate, etc.

[0015] Step four: Each node uses the shared memory module to transfer the perceived communication state information to the intelligent engine, realizing fast data transfer. The process of transferring data using the shared memory module is introduced later.

[0016] Step five: The intelligent engine module on each node starts to work. The specific introduction of this module is described below. After receiving the communication state information perceived by the node, the intelligent engine module makes reasoning and decision according to the existing model or rule.

[0017] Step six: Since data transmission and intelligent engine running in steps three to four require a certain amount of time, a time synchronization module is used to manage the running of the intelligent engine and the time scheduling of the wireless communication module, so that the system time axis remains synchronized.

[0018] Step seven: Each node receives and executes the decision made by the intelligent engine through the shared memory module, changing the surrounding environment state.

[0019] Step eight: Repeat steps three to seven until the simulation ends.

[0020] In steps four and seven, the shared memory module is used for fast data interaction. The specific process is as follows:

[0021] 1) Initialize the shared memory module, set the creator of the memory pool, set the wireless communication environment information that the node can obtain and the controllable communication parameter as the structure of the transmission;

[0022] 2) Bind the structure to be transmitted with the wireless communication module and the intelligent engine module;

[0023] 3) When the environmental state information perceived by the node changes, the wireless communication module writes these information into the shared memory pool;

[0024] 4) The intelligent engine module reads the data in the shared memory pool and makes learning, reasoning and decision, and writes the decision information into the shared memory pool;

[0025] 5) The wireless communication module reads the decision information from the shared memory pool and changes the communication state of the node;

[0026] 6) In order to ensure the consistency of the above data and the safety of access, the semaphore mechanism is used to control the transmission order to ensure that the node wireless communication module and the intelligent engine do not conflict when reading and writing the shared memory module.

[0027] Step five uses the intelligent engine module to provide decision, and its specific workflow is as follows:

[0028] 1) Initialize the storage size of the knowledge base, the network model parameters in the reasoning engine and the learning parameters in the learning engine;

[0029] 2) After receiving the communication state information, the intelligent engine processes the data into a form that can be transmitted into the network model, and transmits it to the reasoning engine for reasoning and decision;

[0030] 3) The intelligent engine stores the existing communication state information and decision information into the knowledge base, which is used to speed up the reasoning and decision of the reasoning engine and provides historical data for the learning engine to update the network model;

[0031] 4) The intelligent engine transmits the historical communication state information and decision information in the knowledge base to the learning engine, updates the network model relied on by reasoning and decision through learning and training of historical data, and realizes the continuous evolution of the intelligent engine.

[0032] The time synchronization module is used in step six to manage the time axis and running of the whole system, and its internal workflow is as follows:

[0033] 1) Record the time stamp before and after each read and write operation of the shared memory, and calculate the time cost of data interaction;

[0034] 2) record timestamps before and after processing the transmitted data, calculate the time cost of data processing;

[0035] 3) when using GPU to accelerate the learning method, the operation of the GPU is usually asynchronous, so the event management mechanism of CUDA is used to synchronize and accurately calculate the time cost of learning;

[0036] 4) calculate the sum of the time cost of steps 1) to 3), and resynchronize the time axis of the wireless communication module and the intelligent engine module.

[0037] The above method is applied to the channel set selected by the participant i, P i The power set selected by the participant i is represented; The goal of the participant in the game is to avoid interference while avoiding the selection of the same channel by a large number of participants during transmission, which causes mutual backoff, so it is a typical application, which will be further described in the embodiments.

[0038] Beneficial effects: the present application provides a joint simulation method and system of intelligent game and communication network, with the help of virtualization and container technology and intelligent engine module, each node can make independent intelligent decision, reduce the resource cost in large-scale simulation scenario, provide fast and scalable node deployment method; With the help of shared memory module, the time cost of data interaction is reduced, which makes up for the deficiency of intelligent wireless communication node in real-time performance; Through the construction of time synchronization module, the synchronization of intelligent decision, data transmission time cost and system running time is ensured, the influence of ignoring learning time on communication result is avoided, and the fidelity of communication result is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a scene diagram of a joint simulation system of intelligent game and communication network

[0040] Figure 2 is a structure diagram of an intelligent wireless communication node

[0041] Figure 3 is a flowchart of a joint simulation scheme of intelligent game and communication network

[0042] Figure 4 is a scene diagram of an embodiment of a joint simulation system of intelligent game and communication network

[0043] Figure 5 is a running schematic diagram of each intelligent wireless communication node divided by time slot

[0044] Figure 6 is a running flowchart of an intelligent engine based on deep reinforcement learning DETAILED DESCRIPTION

[0045] The application will be further described below in connection with the accompanying drawings and specific embodiments. It should be understood, however, that the application can be implemented in various forms and the following some exemplary and non-limiting embodiments shown in the accompanying drawings and described below are not intended to limit the application to the specific embodiments described.

[0046] Figure 1 shows a schematic diagram of a joint simulation scheme of an intelligent game and a communication network according to an embodiment of the application. Figure 4 The embodiment proposes a joint simulation scheme of an intelligent game and a communication network, and the specific steps are as follows:

[0047] Step one: the game set by the embodiment is a communication anti-interference game, and the game scene is set as shown in Figure 4 The game model is constructed as follows:

[0048] 1) Participants in the game, the participants in the game are communication users;

[0049] 2) The strategy set of each participant is S i {C i ,P i}, wherein C i represents the channel set that participant i can choose, and P i represents the power set that participant i can choose;

[0050] 3) The goal of the participants in the game is to avoid interference while avoiding a large number of participants selecting the same channel for transmission, which leads to mutual avoidance. Therefore, the utility function of each participant is defined as:

[0051] U i = βThroughput i

[0052] Wherein Throughput i represents the throughput of participant i, and β is a normalization parameter;

[0053] 4) The game is a static incomplete information game.

[0054] Step two: describe the static incomplete information game problem as a Markov decision process. The communication users select channels for mutual communication transmission, and the interference will select channels for interference according to the interference mode, and the communication users will change their channel selection strategy through the intelligent engine at each communication time slot. The above process is described as a Markov decision process (MDP), and the specific process is as follows:

[0055] 1) The system runs in time slots, as shown in Figure 5 The running schematic diagram of each node according to time slot division is shown in

[0056] 2) Define state, the state of communication user i at time slot t can be expressed as where represents the channel selected by communication user i at time slot t, represents the power selected by communication user i at time slot t, represents the interference power of the current communication user at time slot t when the interference j is selected by the channel, the state space is the set of all states , denoted as S. In the actual simulation, the communication user has 4 choices of channel and power, and the interference has 4 choices of channel and power. When the channel of the interference is different from that of the communication user, the interference power of the communication user is 0, i.e. there are only 5 interference power states. Therefore, the size of the state space is 80;

[0057] 3) Define action, the action of communication user i at time slot t can be expressed as represents the channel and power selected by the user at time slot t+1, and the action space is the set of all actions , denoted as A. In the actual simulation, the communication user has 4 choices of channel and power, i.e. the size of the action space is 16;

[0058] 4) Define state transition function, since the next state is determined by the current action and the change of the interference, the transition function can be defined as J ∈ [0, 1], which represents the probability of transitioning from the current state s t ∈ S to the next state s t by selecting action a t+1 ∈ A and J where A 2 represents the action space of the interference; the probability of the transition function is determined by the action space and the state space, i.e. the probability of changing from a certain state to the next state. According to the size of the state space and the action space above, the probability of selecting any action is and the probability of transitioning from a certain state to the next state is:

[0059]

[0060] In the actual simulation, since the communication user will change its selection strategy through learning, the probability of selecting action is a time-varying value that cannot be calculated;

[0061] 5) Define reward function, when communication user i performs action at state , it will obtain a reward value​ , where SINRitdenotes the signal-to-interference ratio of node i at time slot t is defined as

[0062]

[0063] where σ 2 represents the impact of noise power, and we set the minimum SINR threshold T h during the communication process. denotes a successful communication, otherwise a failed communication. Thus, the reward function is defined as

[0064]

[0065] which is the utility function in the game described above. In the actual simulation, we set the threshold T h = 15 dB.

[0066] 6) Define the system objective, which is to obtain the optimal strategy π such that all nodes in the system are in the equilibrium state of the game, i.e., each node has obtained its optimal strategy to maximize the long-term expected return. Therefore, the optimization problem of the system can be modeled as

[0067]

[0068] where γ (0 < γ ≤ 1) is the discount factor, representing the importance of future returns to current returns, denotes the expectation.

[0069] Step three: build an intelligent engine. In the embodiment, the intelligent engine will be built based on a deep reinforcement learning framework. The running process of the intelligent engine is shown in Figure 6 , and the specific process is as follows:

[0070] 1) Initialize the policy network Q and the target network of the intelligent engine based on the parameters θ and θ- Initialize the cache size of the experience replay for storing the tuple (s t ,a t ,r t ,s t+1 ) containing the state at the current time, the action at the current time, the current reward, and the state at the next time; set the learning rate α of the learning algorithm, the discount factor γ, and the initial exploration rate ∈0. In the embodiment, the cache size of the experience replay is 128, the learning rate α = 1 * 10 -4 , the discount factor γ = 0.99, and the initial exploration rate ∈0= 0.99.

[0071] 2) At each time t, the intelligent engine receives the state s t and the reward rt And select action a according to the ∈–greedy strategy. t :

[0072]

[0073] Where r is a random number between 0 and 1, Q(s) t ,a t ;θ) is determined by the parameter θ in state s t Next action a t The value function, with the exploration rate ∈ following a decay formula:

[0074]

[0075] Where ∈ min For the minimum exploration rate, ∈ decay For the decay of the exploration rate, specifically in the implementation example, ∈ min =0.02, ∈ decay =100.

[0076] 3) (s) t ,a t ,r t ,s t+1 The experience is stored in the experience replay buffer, and a small batch of experiences (s) is randomly sampled from the experience replay buffer. i ,a i ,r i ,s i+1 The target value y is calculated based on the small batch size of the sampled data.

[0077]

[0078] Where 'a' is the network that can... The action with the largest value, where γ is the discount factor. To make the policy network more closely resemble the target network, it is considered a regression problem, and the loss function is minimized using gradient descent. The loss function is the mean squared error function, i.e.:

[0079]

[0080] in, It indicates a desire for the expected value.

[0081] The expected value of the function is calculated in the simulation using the output values ​​of the policy network and the target value at different times; it is not a fixed data point, and the output values ​​of the policy network and the target network are also time-varying. Therefore, it is described using a formula.

[0082] 4) For the target network, set its every C rounds to perform an update, periodically copy the parameters of the policy network to the target network, so as to realize the update of the target network. Specifically in the embodiment, C = 20.

[0083] Step four: initialize the joint simulation system of intelligent game and communication network, at the system level, initialize the total time length of simulation, time slot of decision, number of nodes, and parameters such as geographical position, movement model and node type of each node; at the communication level, initialize the traffic model of application layer, TCP or UDP protocol of transport layer, routing protocol and topology type of network layer, MAC protocol of MAC layer and channel, power and other parameters of physical layer.

[0084] Specifically, at the system level, the total time length of simulation of the embodiment is 200s, each decision time slot is 1s, and the number of nodes is 7, of which 4 nodes are communication users, 1 node is a base station, and 2 nodes are interference. The geographical position distribution of each node is as shown in Figure 3 The movement model adopts a static movement model; at the communication level, the application layer traffic model of the communication user is a constant bit rate traffic model, the transport layer adopts UDP protocol, the routing protocol of the network layer adopts an optimized link state protocol (OLSR), and the topology connection is as shown in Figure 3 The MAC layer adopts CSMA / CA protocol, and the channel and transmission power of the communication node in the physical layer are adjustable parameters. For any communication user i, the center frequency set of the selectable channel is {5.18, 5.22, 5.26, 5.30} GHz, the bandwidth of the channel is 20 MHz, and the selectable transmission power set is {14, 16, 18, 20} dBm; correspondingly, for any interference node j, the selectable channel is consistent with the communication user, and the selectable power set is {10, 12, 14, 16} dBm. The interference has three interference modes of fixed frequency interference, swept frequency interference and following interference. In the fixed frequency interference mode, the center frequencies of the selected channels of the two interference nodes are 5.22 GHz and 5.30 GHz; in the swept frequency interference mode, the initial center frequencies of the channels of the interference nodes are 5.18 GHz and 5.26 GHz, and the next channel in the channel set is scanned in turn according to the time slot; in the following interference mode, the interference nodes randomly select the initial channel, select the channel with the most communication nodes used according to the time slot, and through the synchronization between the interference nodes, it is ensured that the two interference nodes will not select the repeated channel for interference.

[0085] Step five: the communication user communicates with other communication users or base stations according to the business demand, and the joint simulation system of intelligent game and communication network starts to run. The shared memory module, the intelligent engine module and the time synchronization module perform simulation operation according to the steps in the technical content until the simulation time ends.

[0086] The embodiments of the present application are described in detail above with reference to the drawings, but the present application is not limited to the above-described embodiments, and various changes within the knowledge of those skilled in the art are included in the scope of the present application without departing from the spirit of the present application.

Claims

1. A method for joint simulation of an intelligent gaming and communication network, characterized in that, For each node in the communication network, the node is separately deployed by using virtualization and container technology, so that each node has independent and scalable wireless communication module, shared memory module, intelligent engine module and time synchronization module, thereby sharing the same wireless communication environment, performing wireless communication intelligent game and joint simulation of the communication network, and the flow is as follows: Step one: constructing a game model of wireless communication, the specific steps are as follows: 1) determine the participants in the wireless communication game model Γ={1,2,…,n}, the participants are all nodes in the communication network; 2) defining a set of strategies S for each of the participants i including power control, spectrum selection, resource allocation; 3) defining each of said participants a utility function U i including transmission rate, signal to interference ratio, resource utilization; 4) select the type of game, including static game, dynamic game, incomplete information game; Wherein, n represents the number of participants, and subscript i represents any participant; Step two: initialize the joint simulation system of intelligent game and communication network: at the system level, initialize the total simulation time, the number of nodes, and the geographical position, movement model and node type parameters of each node; at the communication level, initialize the traffic model of the application layer, the TCP or UDP protocol of the transport layer, the routing protocol and topology structure of the network layer, the MAC protocol of the MAC layer and the channel and power parameters of the physical layer; at the intelligent level, initialize the knowledge base, reasoning engine and learning engine; according to the above initialization, the wireless communication module will establish the corresponding communication link for each node according to the geographical position and network topology structure of the node; Step three: run the joint simulation system of intelligent game and communication network, the nodes start to communicate and perceive the surrounding communication state information, including routing state, channel state, throughput and packet loss rate; Step four: each node uses the shared memory module to transmit the perceived communication state information to the intelligent engine to realize fast data transmission; the process of transmitting data by using the shared memory module is introduced later; Step five: the intelligent engine module on each node starts to work, the specific introduction of the module is described below, after receiving the communication state information perceived by the node, the module makes reasoning and decision according to the existing model or rule; Step six: because the data transmission in steps three to four and the running of the intelligent engine need a certain time, the time synchronization module is used to manage the running of the intelligent engine and the time scheduling of the wireless communication module, so that the system time axis remains synchronous; Step seven: each node receives and executes the decision made by the intelligent engine through the shared memory module, and changes the surrounding environment state; Step eight: repeat steps three to seven until the simulation is finished; In steps four and seven, the shared memory module is used by the node for fast data interaction, and the specific process is as follows: 1) initialize the shared memory module, set the creation of the memory pool, set the wireless communication environment information and controllable communication parameters that can be obtained by the node to the structure of the transmission; 2) bind the structure to be transmitted with the wireless communication module and the intelligent engine module; 3) when the environment state information perceived by the node changes, the wireless communication module will write these information into the shared memory pool; 4) The intelligent engine module reads the data in the shared memory pool, and learns, reasons and makes decisions, and writes the decision information into the shared memory pool; 5) The wireless communication module reads the decision information from the shared memory pool, and changes the communication state of the node; 6) In order to ensure the consistency of the above data and the safety of access, the semaphore mechanism is used to control the transmission sequence, so that the wireless communication module and the intelligent engine of the node do not conflict when reading and writing the shared memory module.

2. The method of claim 1, wherein the method further comprises: The intelligent engine module used in step five is used to provide decisions, and the process is as follows: 1) Initialize the storage size of the knowledge base, the network model parameters in the reasoning engine, and the learning parameters in the learning engine; 2) After receiving the communication state information, the intelligent engine processes the data into a form that can be transmitted into the network model, and transmits it to the reasoning engine for reasoning and decision-making; 3) The intelligent engine stores the existing communication state information and decision information in the knowledge base, which is used to speed up the reasoning and decision-making speed of the reasoning engine, and provides historical data for the learning engine to update the network model; 4) The intelligent engine transmits the historical communication state information and decision information in the knowledge base to the learning engine, updates the network model relied on by reasoning and decision-making through learning and training of historical data, and realizes the continuous evolution of the intelligent engine.

3. The method of claim 1, wherein the method further comprises: The time synchronization module is used in step six to manage the time axis and running of the entire system, and the internal workflow is as follows: 1) Record the time stamp before and after each read and write operation of the shared memory, and calculate the time cost of data interaction; 2) Record the time stamp before and after processing the transmitted data, and calculate the time cost of data processing; 3) When using GPU to accelerate the learning method, the operation of GPU is asynchronous, so the event management mechanism of CUDA is used to synchronize and accurately calculate the time cost of learning; 4) Calculate the sum of the time costs of steps 1) to 3), and resynchronize the time axis of the wireless communication module and the intelligent engine module.

4. The intelligent game and communication network joint simulation method according to claim 1, characterized in that, Step one: construction of game model: 1) The set of strategies of a player in a game is S i {C i ,P i} where C i represents the set of channels chosen by player i and P i represents the set of powers chosen by player i. 2) The goal of the participants in this game is to avoid interference while trying to avoid a large number of participants choosing the same channel for transmission, resulting in mutual avoidance, so the utility function of each participant is defined as: U i = β Throughput i where Throughput i represents the throughput of participant i, and β is a normalization parameter. 3) The game is a static incomplete information game; Step two: describe the static incomplete information game problem as a Markov decision process; communication users select channels for mutual communication transmission, and interference will select channels for interference according to the interference mode, and communication users will change their channel selection strategy through intelligent engines at each communication time slot; the above process is constructed into a Markov decision process MDP for description, and the process is as follows: 1) The system runs in time slots, and the running schematic diagram of each node is divided by time slots; 2) Define the state: The state of communication user i in time slot t is represented as follows: in This indicates the channel selected by communication user i at time slot t. This represents the power selected by communication user i in time slot t. This indicates that the current communication user is selecting a channel. The interference power of the disturbance j is represented in the state space as S; 3) define actions, the action of the user i at time slot t is denoted as denoting the channel and power selected by the user at time slot t+1, the action space is denoted as A; 4) define the state transition function, since the next state is determined by the current action and the change of disturbance, the transition function is defined as ρ = S x S x A x A J ∈ [0,1], represents the probability of choosing action a t ∈ A at the current state s t ∈ S and transferring to the next state s t+1 , where A J represents the action space of the disturbance; 5) define a reward function, which the communication user i will get when performing action at state at time Here the signal-to-interference ratio of node i at time slot t is defined as: ​ Where σ 2 To represent the impact of noise power, the minimum SINR threshold for actual communication is set to T. h ,when A successful communication indicates successful communication; otherwise, communication fails. Therefore, the reward function is defined as follows: This is the utility function in the above game; 6) define system target, the system target is to get the optimal strategy π, make all nodes in the system in the equilibrium state of game, that is, each node has got its own optimal strategy to maximize the long-term expected return, therefore, the optimization problem of the system is modeled as: where γ (0 < γ < 1) is a discount factor that represents the importance of future rewards over current rewards, denotes the expectation; Step three: build intelligent engine, the intelligent engine will be based on deep reinforcement learning framework to build; the running process of the intelligent engine is: 1) based on the parameters θ and θ - Initialize the policy network Q and the target network of the intelligent engine Initialize the cache size of the experience replay for storing the tuples (s t ,a t ,r t ,s t+1 ) containing the current time state, the current time action, the current reward and the next time state; set the learning rate α, the discount factor γ and the initial exploration rate ∈0 of the learning algorithm; the experience replay cache size is 128, the learning rate α = 1*10 -4 -3, the discount factor γ = 0.99 and the initial exploration rate ∈0 = 0.99; 2) At each time instant t, the intelligent engine receives the state s incoming from the wireless communication simulation system t and the reward r for the current time instant t and selects an action a according to the ∈-greedy policy t : where r is a random number between 0 and 1, Q(s t ,a t ) is the value function of action a t at state s t determined by parameter θ, and exploration rate ∈ follows the decay formula: where ∈ min is the minimum value of the exploration rate, ∈ decay is the decay of the exploration rate; 3) store (s t ,a t ,r t ,s t+1 ) in a replay buffer and randomly sample a mini-batch of experiences (s i ,a i ,r i ,s i+1 ) from the replay buffer; compute target values y from the sampled mini-batch; where a is the action that maximizes the target network value, and g is the discount factor; in order to make the policy network closer to the target network, consider it as a regression problem, use the gradient descent method to minimize its loss function, and use the mean square error function, that is: wherein denotes the expectation; 4) for the target network, set it to update once every C rounds, periodically copy the parameters of the strategy network to the target network, so as to realize the update of the target network; Step four: initialize the joint simulation system of intelligent game and communication network, in the system level, initialize the total time length of simulation, time slot of decision, number of nodes, and geographical position, movement model and node type parameters of each node; In the communication level, initialize the traffic model of the application layer, the TCP or UDP protocol of the transport layer, the routing protocol and topology type of the network layer, the MAC protocol of the MAC layer and the channel and power parameters of the physical layer; Interference has three interference modes of fixed frequency interference, sweep frequency interference and follow-up interference; Communication users communicate with other communication users or base stations according to business needs, the joint simulation system of intelligent game and communication network starts to run, the shared memory module, the intelligent engine module and the time synchronization module run according to the above steps, until the simulation time is over.

Images