Joint simulation method and system for intelligent game and communication network

Abstract

A joint simulation method of an intelligent game and a communication network uses virtualization and container technologies to independently deploy nodes, so that each node has a wireless communication module, a shared memory module, an intelligent engine module and a time synchronization module which are mutually independent and expandable, and comprises the following steps: 1) constructing a game model of wireless communication; 2) initializing a joint simulation system of an intelligent game and a communication network; 3) operating a joint simulation system of an intelligent game and a communication network, and enabling the nodes to start communication transmission and perceive surrounding communication state information, such as a routing state, a channel state, throughput, packet loss rate and the like; 4) transmitting sensed communication state information to an intelligent engine by each node by using a shared memory module to realize rapid data transmission; and 5) the intelligent engine module on each node starts to work, and performs reasoning and decision making according to the existing model or rule after receiving the communication state information sensed by the node.

Description

Technical Field

[0001] The invention belongs to the field of wireless communication network simulation, and in particular relates to a joint simulation method of intelligent game and communication network. Background Art

[0002] The joint simulation of intelligent games and communication networks is a comprehensive simulation method that integrates intelligent decision-making processes and communication network characteristics. It aims to evaluate and optimize the effect of game strategies in actual communication network environments. Specifically, each communication node acts as a player and, under limited communication resource conditions, improves communication indicators such as resource utilization and throughput through strategy selection. During the game, each communication node adjusts its strategy based on its own observation information so that the node finds a balance in the local optimum or the global optimum.

[0003] Common communication network simulation tools include OMNet++, OPNET and NS-3, which can simulate different types of networks and real communication processes; while the research on intelligent games mainly focuses on combining traditional game theory with intelligent algorithms such as machine learning and reinforcement learning to solve problems such as resource competition and strategy optimization. With the development of technologies such as 5G, 6G networks and the Internet of Vehicles, the demand for research on collaboration and confrontation issues in multi-user and complex environments has increased significantly. The existing research on the joint simulation of intelligent games and communication networks still has the following problems in practical applications: 1) The simulation verification of the joint intelligent game and communication network is mainly focused on the behavioral level, and there is a lack of simulation that combines the behavioral level with the signal level, so the fidelity of the method cannot be verified; 2) For the joint simulation of intelligent games and communication networks on a larger scale, a large number of communication nodes and complex communication environments need to be simulated. Each time the node information is processed and the decision is made, it will bring huge performance overhead. However, many studies do not consider the system overhead of signal-level simulation in large-scale scenarios, resulting in limited simulation scale and insufficient scalability; 3) In multi-agent scenarios and dynamic communication environments, the learning and decision-making of intelligent games are difficult to generate in real time, and most current studies do not consider the limitation of computing power and ignore the real-time nature of the algorithm, so it is difficult to apply it to wireless communication networks with dynamically changing environments.

[0004] CN201810458873.5 discloses a method for detecting abnormal users in a communication network based on a generative adversarial network. First, the data of abnormal users is regularized to obtain data with consistent dimension and magnitude; the generative adversarial network is trained, that is, the generator and the discriminator are trained to achieve oversampling of abnormal users; the data generated by the generator and the normal users form a training data set, and the training data set is classified using a deep fully connected neural network to determine the user type; through the mutual game-style training method between the neural networks in the generative adversarial network, the distribution of abnormal user data is approximated, abnormal user detection is achieved, and the technical problem that the imbalanced proportion in the training data set affects the training effect of the classification model is solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the problems of limited system resources, poor scalability and low real-time performance of the existing intelligent game and communication network joint simulation methods when facing large-scale and dynamic communication environments. The present invention proposes an intelligent game and hungry communication network joint simulation method and system, which combines communication with intelligence. On the basis of the communication network module, the nodes are extracted from the communication network simulation system with the help of virtualization technology, and an intelligent engine module is constructed so that each node can make autonomous intelligent decisions independently of each other, thereby improving the scalability of the system; a shared memory module and a time synchronization module are constructed to achieve rapid interaction between the wireless communication module and the intelligent engine module, and the effects of computing power limitations and time overhead are calculated in the actual communication system, thereby improving the operating efficiency of the method and the fidelity of the system.

[0006] To achieve the above purpose, the technical solution proposed in the present invention is: a joint simulation method of intelligent game and communication network, for each node in the communication network, the node is deployed separately by using virtualization and container technology, so that each node has an independent and scalable wireless communication module, a shared memory module, an intelligent engine module and a time synchronization module, and shares the same wireless communication environment. The scenario of the joint simulation system of intelligent game and communication network is as follows Figure 1 As shown in Figure 2, the structure of the intelligent communication node is as follows: Figure 2 The overall process of the solution is as follows:

[0007] Step 1: Construct a wireless communication game model. The specific steps are as follows:

[0008] 1) Determine the participants Γ={1,2,…,n} in the wireless communication game model, where the participants are all nodes in the communication network;

[0009] 2) Define the strategy set S of each participant i , such as power control, spectrum selection, resource allocation, etc.;

[0010] 3) Define the utility function U for each participant i , such as transmission rate, signal-to-interference ratio, resource utilization, etc.;

[0011] 4) Select the type of game, such as static game, dynamic game, incomplete information game, etc.

[0012] Here, n represents the number of participants and the subscript i represents any participant.

[0013] Step 2: Initialize the joint simulation system of intelligent game and communication network, such as Figure 1 As shown. At the system level, initialize the total simulation duration, the number of nodes, and the geographical location, mobility model, and node type of each node; at the communication level, initialize the traffic model of the application layer, the TCP or UDP protocol of the transport layer, the routing protocol and topology of the network layer, the MAC protocol of the MAC layer, and the channel, power, and other parameters of the physical layer; at the intelligent level, initialize the knowledge base, reasoning engine, and learning engine. Based on the above initialization, the wireless communication module will establish a corresponding communication link for each node according to the geographical location and network topology of the node.

[0014] Step 3: Run the joint simulation system of intelligent game and communication network. The nodes start communication transmission and perceive the surrounding communication status information, such as routing status, channel status, throughput, packet loss rate, etc.

[0015] Step 4: Each node uses the shared memory module to transmit the perceived communication status information to the intelligent engine to achieve fast data transmission. The process of using the shared memory module to transmit data will be introduced later.

[0016] Step 5: The intelligent engine module on each node starts working. The specific introduction of this module is described below. After receiving the communication status information perceived by the node, it makes inference decisions based on the existing model or rules.

[0017] Step 6: Since the data transmission and intelligent engine operation in step 3 to step 4 require a certain amount of time, the time synchronization module is used to manage the operation of the intelligent engine and the time scheduling of the wireless communication module to keep the system time axis synchronized.

[0018] Step 7: Each node receives and executes the decision made by the intelligent engine through the shared memory module to change the state of the surrounding environment.

[0019] Step 8: Repeat steps 3 to 7 until the simulation ends.

[0020] In steps 4 and 7, the nodes use the shared memory module for fast data interaction. The specific process is as follows:

[0021] 1) Initialize the shared memory module, set the creator of the memory pool, and set the wireless communication environment information and adjustable communication parameters that can be obtained by the node communication status information point perception as the structure during transmission;

[0022] 2) Bind the structure to be transmitted with the wireless communication module and the intelligent engine module;

[0023] 3) When the environmental status information perceived by the node changes, the wireless communication module will write this information into the shared memory pool;

[0024] 4) The intelligent engine module reads the data in the shared memory pool, performs learning, reasoning and decision-making, and writes the decision information into the shared memory pool;

[0025] 5) The wireless communication module reads the decision information from the shared memory pool and changes the communication status of the node;

[0026] 6) In order to ensure the consistency of the above data and the security of access, the semaphore mechanism is used to control the transmission order to ensure that there is no conflict between the node wireless communication module and the intelligent engine when reading and writing the shared memory module.

[0027] Step 5 uses the intelligent engine module to provide decision-making. The specific workflow is as follows:

[0028] 1) Initialize the storage size of the knowledge base, the network model parameters in the inference engine, and the learning parameters in the learning engine;

[0029] 2) After receiving the communication status information, the intelligent engine processes the data into a form that can be passed into the network model and then passed to the inference engine for inference decision making;

[0030] 3) The intelligent engine stores the existing communication status information and decision information in the knowledge base, which is then provided to the reasoning engine to accelerate the reasoning decision-making speed, and provides the learning engine with historical data for updating the network model;

[0031] 4) The intelligent engine passes the historical communication status information and decision information in the knowledge base to the learning engine, and updates the network model on which the reasoning decision relies through learning and training of historical data, thereby realizing the continuous evolution of the intelligent engine.

[0032] In step 6, the time synchronization module is used to manage the timeline and operation of the entire system. Its internal workflow is as follows:

[0033] 1) Record the timestamp before and after each shared memory read and write operation, and calculate the time overhead of data interaction;

[0034] 2) Record the timestamps before and after processing the transmitted data and calculate the time overhead used for data processing;

[0035] 3) When using GPU to accelerate learning methods, the operation of GPU is usually asynchronous, so CUDA's event management mechanism is used to synchronize and accurately calculate the time overhead of learning;

[0036] 4) Calculate the sum of the time overheads used in steps 1) to 3), and resynchronize the time axes of the wireless communication module and the intelligent engine module.

[0037] The above method is applied to the channel set selected by participant i, P i represents the power set selected by participant i; the goal of the participants in the game is to avoid interference while trying to avoid a large number of participants choosing the same channel for transmission during transmission, which causes mutual backoff. This is a typical application and will be further described in detail in the embodiments.

[0038] Beneficial effects: The present invention provides a joint simulation method and system for intelligent games and communication networks. With the help of virtualization and container technology and an intelligent engine module, each node can make intelligent decisions independently, reducing resource overhead in large-scale simulation scenarios and providing a fast and scalable node deployment method; with the help of a shared memory module, the time overhead during data interaction is reduced, making up for the lack of real-time performance of intelligent wireless communication nodes; through the construction of a time synchronization module, the synchronization of the time overhead of intelligent decision-making and data transmission with the system running time is ensured, avoiding the impact of ignoring the learning time on the communication results, and significantly improving the fidelity of the communication results. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the joint simulation system scenario of intelligent game and communication network.

[0040] Figure 2 This is the structure diagram of the intelligent wireless communication node

[0041] Figure 3 This is a flowchart of the joint simulation solution of intelligent game and communication network.

[0042] Figure 4 This is a schematic diagram of the scenario of the joint simulation system of intelligent game and communication network.

[0043] Figure 5 This is a schematic diagram of the operation of each intelligent wireless communication node divided by time slot

[0044] Figure 6 This is the operation flow chart of the intelligent engine based on deep reinforcement learning DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. However, it should be understood that the present invention can be implemented in various forms, and some exemplary and non-limiting embodiments shown in the accompanying drawings and described below are not intended to limit the present invention to the specific embodiments described.

[0046] Attached Figure 4 The figure is a schematic diagram of a scenario of a joint simulation solution of intelligent game and communication network proposed in this embodiment, and the specific steps are as follows:

[0047] Step 1: The game set in the embodiment is a communication anti-interference game, and the game scene is set as follows: Figure 4 As shown, the construction of the game model is as follows:

[0048] 1) Participants in the game, where the participants in the game are communication users;

[0049] 2) Each participant’s strategy set is S i {C i ,P i}, where C i represents the set of channels that participant i can choose, P i represents the power set that participant i can choose;

[0050] 3) The goal of the participants in this game is to avoid interference while trying to avoid a large number of participants choosing the same channel to transmit, which will cause mutual retreat. Therefore, the utility function of each participant is defined as:

[0051] U i =βThroughput i

[0052] Throughput i represents the throughput of participant i, β is the normalization parameter;

[0053] 4) This game is a static incomplete information game.

[0054] Step 2: Describe the static incomplete information game problem as a Markov decision process. Communication users select channels for mutual communication transmission, interference will select channels for interference according to the interference pattern, and communication users will change their channel selection strategies through the intelligent engine in each communication time slot. The above process is constructed as a Markov decision process (MDP) for description, and the specific process is as follows:

[0055] 1) The system runs in time slots, such as Figure 5 Shown is a schematic diagram of the operation of each node divided by time slot;

[0056] 2) Define the state. The state of communication user i at time slot t can be expressed as in represents the channel selected by communication user i at time slot t, represents the power selected by communication user i at time slot t, Indicates that the current communication user is selecting a channel The interference power of interference j is received at the time, and the state space is all states The set is denoted as S. In the actual simulation, the communication user has 4 optional channels and transmission powers, and the interference has 4 optional channels and powers. When the interference channel is different from the communication user, the interference power received by the communication user is 0, that is, there are only 5 interference power states. It can be seen that the size of the state space is 80;

[0057] 3) Define the action. The action of communication user i at time slot t can be expressed as represents the channel and power selected by the user at time slot t+1, and the action space is all actions The set is denoted as A. In the actual simulation, the communication user has 4 choices for the optional channels and transmission power, that is, the size of the action space is 16;

[0058] 4) Define the state transfer function. Since the state at the next moment is determined by the current action and the change of interference, the transfer function can be defined as ρ = S × S × A × A J ∈[0,1], indicating that in the current state s t ∈S select action a t ∈A and Transition to next state s t+1 The probability of A J represents the action space of interference; the probability of the transfer function is determined by the action space and the state space, that is, the probability of changing from a certain state to the next state. According to the size of the state space and action space above, it can be seen that when choosing any action The probability of When , the probability of transitioning from one state to the next state is:

[0059]

[0060] In the actual simulation, since the communication users will change their selection strategies through learning, the selection action Probability is a time-varying value that cannot be calculated;

[0061] 5) Define the reward function. When the communication user i is in state Execute actions when When , where the signal-to-interference ratio of node i at time slot t is defined as for:

[0062]

[0063] where σ 2 Indicates the impact of noise power. We set the minimum SINR threshold during communication to T h ,when When , it indicates that the communication is successful, otherwise the communication fails. Therefore, the reward function is defined as:

[0064]

[0065] This is the utility function in the above game. In the actual simulation, the threshold T is set h =15dB.

[0066] 6) Define the system goal. The system goal is to obtain the optimal strategy π so that all nodes in the system are in a state of game equilibrium, that is, each node has its own optimal strategy to maximize the long-term expected return. Therefore, the optimization problem of the system can be modeled as:

[0067]

[0068] Among them, γ (0<γ≤1) is the discount factor, which indicates the importance of future returns to current returns. Expressing expectation.

[0069] Step 3: Build an intelligent engine. In the embodiment, the intelligent engine will be built based on the deep reinforcement learning framework. The operation process of the intelligent engine is as follows: Figure 6 As shown, the specific process is:

[0070] 1) Initialize the policy network Q and target network of the intelligent engine based on parameters θ and θ Initialize the size of the experience replay buffer, which is used to store tuples containing the current state, current action, current reward, and next state (s t ,a t ,r t ,s t+1 ); Set the learning rate α, discount factor γ, and initial exploration rate ∈ 0 of the learning algorithm. Specifically, in the embodiment, the experience replay buffer size is 128, and the learning rate α = 1*10 -4 , discount factor γ = 0.99, initial exploration rate ∈ 0 = 0.99;

[0071] 2) At each time t, the intelligent engine receives the state s from the wireless communication simulation system t and the current reward rt , and select action a according to the ∈–greedy strategy t :

[0072]

[0073] Among them, r is a random number between 0 and 1, Q(s t ,a t ; θ) is determined by the parameter θ in state s t Next action a t The value function of , the exploration rate ∈ follows the following decay formula:

[0074]

[0075] where ∈ min is the minimum value of the exploration rate, ∈ decay is the attenuation of the exploration rate. Specifically, ∈ min =0.02,∈ decay =100.

[0076] 3) will (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer, and a small batch of experience (s) is randomly sampled from the experience replay buffer i ,a i ,r i ,s i+1 ). Calculate the target value y based on the sampled mini-batch:

[0077]

[0078] Among them, a is the target network The action with the largest value, γ is the discount factor. In order to make the policy network closer to the target network, it is considered as a regression problem, and the gradient descent method is used to minimize its loss function. The loss function uses the mean square error function, that is:

[0079]

[0080] in, Expressing expectation.

[0081] The expected value of the function is calculated by the output value and target value of the policy network at different times in the simulation. It is not a fixed data. The output value of the policy network and the target network is also time-varying. Therefore, it is described by a formula.

[0082] 4) For the target network, it is set to be updated once every C rounds, and the parameters of the policy network are periodically copied to the target network, so as to achieve the update of the target network. Specifically in the embodiment, C=20.

[0083] Step 4: Initialize the joint simulation system of intelligent game and communication network. At the system level, initialize the total simulation time, decision time slot, number of nodes, geographical location of each node, mobility model, node type and other parameters; at the communication level, initialize the traffic model of the application layer, TCP or UDP protocol of the transport layer, routing protocol and topology type of the network layer, MAC protocol of the MAC layer and channel, power and other parameters of the physical layer.

[0084] Specifically, at the system level, the total simulation duration of this embodiment is 200s, each decision time slot is 1s, the number of nodes is 7, of which 4 nodes are communication users, 1 node is a base station, and 2 nodes are interference. The geographical location distribution of each node is as follows: Figure 3 As shown in the figure, the mobile model adopts the stationary mobile model; at the communication level, the application layer traffic model of the communication user is a constant bit rate traffic model, the transport layer adopts the UDP protocol, the network layer routing protocol adopts the Optimized Link State Protocol (OLSR), and the topology connection is as follows Figure 3 As shown in the figure, the MAC layer adopts the CSMA / CA protocol, and the channels and transmission powers of the communication nodes in the physical layer are adjustable parameters. For any communication user i, the center frequency set of its optional channels is {5.18, 5.22, 5.26, 5.30} GHz, the bandwidth of the channel is 20 MHz, and the optional transmission power set is {14, 16, 18, 20} dBm; correspondingly, for any interference node j, its optional channels are consistent with the communication users, and its optional power set is {10, 12, 14, 16} dBm. The interference has three interference modes: fixed-frequency interference, swept-frequency interference and following interference. In the fixed-frequency interference mode, the center frequencies of the channels selected by the two interference nodes are 5.22GHz and 5.30GHz respectively; in the swept-frequency interference mode, the center frequencies of the initial channels of the interference nodes are 5.18GHz and 5.26GHz, and the next channel in the channel set is scanned in sequence according to the time slot; in the following interference mode, the interference node randomly selects the initial channel and selects the channel used by the most communication nodes according to the time slot, and through synchronization between the interference nodes, it is ensured that the two interference nodes will not choose repeated channels for interference.

[0085] Step 5: The communication user communicates with other communication users or base stations in the surrounding area according to business needs, and the joint simulation system of intelligent game and communication network starts to run. The shared memory module, intelligent engine module and time synchronization module perform simulation according to the steps in the technical content until the simulation time ends.

[0086] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments, and various changes are also included within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose.

Claims

1. A joint simulation method of intelligent game and communication network, characterized in that: For each node in the communication network, virtualization and container technology are used to deploy the nodes separately, so that each node has an independent and scalable wireless communication module, shared memory module, intelligent engine module and time synchronization module, so as to share the same wireless communication environment and conduct joint simulation of wireless communication intelligent game and communication network. The process is as follows: Step 1: Construct a game model for wireless communication. The specific steps are as follows: 1) Determine the participants Γ={1,2,…,n} in the wireless communication game model, where the participants are all nodes in the communication network; 2) Define the strategy set S of each participant i , such as power control, spectrum selection, resource allocation, etc.; 3) Define the utility function U for each participant i , such as transmission rate, signal-to-interference ratio, resource utilization, etc.; 4) Select the type of game, such as static game, dynamic game, incomplete information game, etc.; Here, n represents the number of participants, and the subscript i represents any participant; Step 2: Initialize the joint simulation system of intelligent game and communication network: at the system level, initialize the total simulation time, the number of nodes, and the geographical location, mobility model and node type of each node; at the communication level, initialize the traffic model of the application layer, the TCP or UDP protocol of the transport layer, the routing protocol and topology of the network layer, the MAC protocol of the MAC layer and the channel, power and other parameters of the physical layer; at the intelligent level, initialize the knowledge base, reasoning engine and learning engine; based on the above initialization, the wireless communication module will establish a corresponding communication link for each node according to the geographical location and network topology of the node; Step 3: Run the joint simulation system of intelligent game and communication network, the nodes start communication transmission and perceive the surrounding communication status information, such as routing status, channel status, throughput, packet loss rate, etc. Step 4: Each node uses the shared memory module to transmit the perceived communication status information to the intelligent engine to achieve fast data transmission; the process of using the shared memory module to transmit data will be introduced later; Step 5: The intelligent engine module on each node starts working. The specific introduction of this module is described below. After receiving the communication status information perceived by the node, it makes inference decisions based on the existing model or rules; Step 6: Since the data transmission and intelligent engine operation in step 3 to step 4 require a certain amount of time, the time synchronization module is used to manage the operation of the intelligent engine and the time scheduling of the wireless communication module so that the system time axis remains synchronized; Step 7: Each node receives and executes the decision made by the intelligent engine through the shared memory module to change the state of the surrounding environment; Step 8: Repeat steps 3 to 7 until the simulation ends.

2. The method for joint simulation of intelligent game and communication network according to claim 1, characterized in that: In steps 4 and 7, the nodes use the shared memory module for fast data exchange. The specific process is as follows: 1) Initialize the shared memory module, set the creator of the memory pool, and set the wireless communication environment information and adjustable communication parameters that can be obtained by the node communication status information point perception as the structure during transmission; 2) Bind the structure to be transmitted with the wireless communication module and the intelligent engine module; 3) When the environmental status information perceived by the node changes, the wireless communication module will write this information into the shared memory pool; 4) The intelligent engine module reads the data in the shared memory pool, performs learning, reasoning and decision-making, and writes the decision information into the shared memory pool; 5) The wireless communication module reads the decision information from the shared memory pool and changes the communication status of the node; 6) In order to ensure the consistency of the above data and the security of access, the semaphore mechanism is used to control the transmission order to ensure that there is no conflict between the node wireless communication module and the intelligent engine when reading and writing the shared memory module.

3. The method for joint simulation of intelligent game and communication network according to claim 1, characterized in that: The intelligent engine module used in step 5 is used to provide decision-making. The process is as follows: 1) Initialize the storage size of the knowledge base, the network model parameters in the inference engine, and the learning parameters in the learning engine; 2) After receiving the communication status information, the intelligent engine processes the data into a form that can be passed into the network model and then passed to the inference engine for inference decision making; 3) The intelligent engine stores the existing communication status information and decision information in the knowledge base, which is then provided to the reasoning engine to accelerate the reasoning decision-making speed, and provides the learning engine with historical data for updating the network model; 4) The intelligent engine passes the historical communication status information and decision information in the knowledge base to the learning engine, and updates the network model on which the reasoning decision relies through learning and training of historical data, thereby realizing the continuous evolution of the intelligent engine.

4. The method for joint simulation of intelligent game and communication network according to claim 1, characterized in that: In step 6, the time synchronization module is used to manage the timeline and operation of the entire system. Its internal workflow is as follows: 1) Record the timestamp before and after each shared memory read and write operation, and calculate the time overhead of data interaction; 2) Record the timestamps before and after processing the transmitted data and calculate the time overhead used for data processing; 3) When using GPU to accelerate learning methods, the operation of GPU is usually asynchronous, so CUDA's event management mechanism is used to synchronize and accurately calculate the time overhead of learning; 4) Calculate the sum of the time overheads used in steps 1) to 3), and resynchronize the time axes of the wireless communication module and the intelligent engine module.

5. The method for joint simulation of intelligent game and communication network according to claim 1, characterized in that the steps 1. Construction of game model: 1) The strategy set of the participants in the game is S i {C i ,P i }, where C i represents the channel set selected by participant i, P i represents the power set selected by participant i; 2) The goal of the participants in this game is to avoid interference while trying to avoid a large number of participants choosing the same channel to transmit, which will cause mutual retreat. Therefore, the utility function of each participant is defined as: U i =βThroughput i Throughput i represents the throughput of participant i, β is the normalization parameter; 3) The game is a static incomplete information game; Step 2: Describe the static incomplete information game problem as a Markov decision process; communication users select channels for mutual communication transmission, interference will select channels for interference according to the interference pattern, and communication users will change their channel selection strategies through the intelligent engine in each communication time slot; the above process is constructed into a Markov decision process (MDP) for description, the process is as follows: 1) The system runs in time slots, and the operation diagram of each node is divided by time slots; 2) Define the state. The state of communication user i at time slot t can be expressed as in represents the channel selected by communication user i at time slot t, represents the power selected by communication user i at time slot t, Indicates that the current communication user is selecting a channel The interference power of interference j is expressed as S in the state space; 3) Define the action. The action of communication user i at time slot t can be expressed as represents the channel and power selected by the user at time slot t+1, and the action space is denoted as A; 4) Define the state transfer function. Since the state at the next moment is determined by the current action and the change of interference, the transfer function can be defined as ρ = S × S × A × A J ∈[0,1], indicating that in the current state s t ∈S select action a t ∈A and Transition to next state s t+1 The probability of A J represents the action space of the disturbance; 5) Define the reward function. When the communication user i is in state Execute action when When , where the signal-to-interference ratio of node i at time slot t is defined as for: where σ 2 Indicates the impact of noise power, and sets the minimum SINR threshold for actual communication to T h ,when ≥T h When , it indicates that the communication is successful, otherwise the communication fails; thus, the reward function is defined as: This is the utility function in the above game; 6) Define the system goal. The system goal is to obtain the optimal strategy π so that all nodes in the system are in a state of game equilibrium, that is, each node has its own optimal strategy to maximize the long-term expected return. Therefore, the optimization problem of the system is modeled as: Among them, γ (0<γ≤1) is the discount factor, which indicates the importance of future returns to current returns. Expressing hope; Step 3: Build an intelligent engine. The intelligent engine will be built based on the deep reinforcement learning framework. The operation process of the intelligent engine is as follows: 1) Initialize the policy network Q and target network of the intelligent engine based on parameters θ and θ Initialize the size of the experience replay buffer, which is used to store tuples containing the current state, current action, current reward, and next state (s t ,a t ,r t ,s t+1 ); Set the learning rate α, discount factor γ, and initial exploration rate ∈ 0 of the learning algorithm; Specifically for the embodiment, the experience replay buffer size is 128, and the learning rate α = 1*10 -4 , discount factor γ = 0.99, initial exploration rate ∈ 0 = 0.99; 2) At each time t, the intelligent engine receives the state s transmitted by the wireless communication simulation system t and the current reward r t , and select action a according to the ∈–greedy strategy t : Among them, e is a random number between 0 and 1, Q(s t ,a t ; θ) is determined by the parameter θ in state s t Next action a t The value function of , the exploration rate ∈ follows the following decay formula: where ∈ min is the minimum value of the exploration rate, ∈ decay is the decay of the exploration rate; 3) will (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer, and a small batch of experience (s) is randomly sampled from the experience replay buffer i ,a i ,r i ,s i+1 ); Calculate the target value y based on the sampled mini-batch: Among them, a is the target network The action with the largest value, γ is the discount factor; in order to make the policy network closer to the target network, consider it as a regression problem, and use the gradient descent method to minimize its loss function. The loss function uses the mean square error function, that is: in, Expressing hope; 4) For the target network, it is set to be updated once every C rounds, and the parameters of the policy network are periodically copied to the target network, so as to update the target network; Step 4: Initialize the joint simulation system of intelligent game and communication network. At the system level, initialize the total simulation time, decision time slot, number of nodes, geographical location, mobility model and node type parameters of each node; at the communication level, initialize the traffic model of the application layer, TCP or UDP protocol of the transport layer, routing protocol and topology type of the network layer, MAC protocol of the MAC layer and channel and power parameters of the physical layer; the interference has three interference modes: fixed frequency interference, sweep frequency interference and follow-up interference; The communication user communicates with other communication users or base stations in the surrounding area according to business needs, and the joint simulation system of intelligent game and communication network starts to run. The shared memory module, intelligent engine module and time synchronization module perform simulation according to the above steps until the simulation time ends.

Images