A Wireless Transmission Method for Distributed Internet of Things Systems Based on Reinforcement Learning

Abstract

This invention discloses a wireless transmission method for a distributed Internet of Things (IoT) system based on reinforcement learning. Age of information (AoI) and energy consumption are used as metrics. Each device is independent, monitoring vehicle location information and transmitting it to the base station. Due to limited wireless resources, only some devices can transmit. Therefore, the reinforcement learning-based IPPO algorithm is used to decide which devices can transmit. The IPPO algorithm can reuse sampled data multiple times, solving the problem of low sample utilization, reducing computational time complexity, and improving device transmission efficiency.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology and uses reinforcement learning methods to solve the wireless transmission problem in distributed Internet of Things (IoT) systems. Background Technology

[0002] With the advancement of technology and the development of various software and hardware, the concept of intelligent driving has become increasingly popular, and new car manufacturers are constantly emerging. Both traditional automakers and internet companies are developing intelligent driving technology from different perspectives. Currently, the industry generally classifies intelligent driving into six levels (Level 0-Level 5) according to the SAE classification system, with higher numbers indicating greater intelligence. Level 0 can be described as having no intelligent functions and relying entirely on the driver; Level 1 is what is commonly referred to as basic driving assistance, where the system can perform some simple driving assistance tasks, such as lane departure warning and forward collision warning; Level 2 is partial automation and is currently the main level at which automakers are implementing intelligent driving technology, where the system can perform tasks such as automatic following and automatic braking; Level 3 is the dividing line of intelligent driving, where the system can autonomously complete all driving tasks in all scenarios, but the driver needs to be in the driver's seat ready to take over at any time to deal with unpredictable risks; Level 4 is currently considered by the industry to be the most promising level of intelligence, that is, achieving highly automated driving in scenarios where external unpredictable factors are almost controllable, such as in fixed patrol routes in parks, where no driver intervention is required and the system completes all driving operations completely autonomously; Level 5 is true driverless driving, which is the highest goal pursued by intelligent driving companies, and no company has yet achieved it, even companies like Google, which have been deeply involved in the field of intelligent driving for decades, are still only in the testing phase.

[0003] Tracking is a classic problem in computer vision, and vehicle tracking is crucial in intelligent driving scenarios. It not only compensates for missed detections caused by purely visual inspection but also depicts the target's trajectory. Furthermore, with the development of 5G technology and the emergence of diverse services, various quality of service (QoS) requirements are being placed on existing communication networks. As a key enabling technology and infrastructure of the digital economy era, 5G's services have expanded from simple mobile communication to ubiquitous connectivity and application scenarios. Remote control technology, as a vital technology in production and daily life, is gaining increasing attention. In modern enterprise production, many parameters on the production site require real-time monitoring and supervision, necessitating remote control. Currently, there are two main types of remote control: one combines on-site and remote monitoring, and the other involves data collection and processing at a remote computer when there is no on-site monitoring system. Both of these remote control technologies require direct information transmission between the controlled equipment and the controlled site. Information transmission can be categorized into wired and wireless transmission methods. Wireless transmission refers to data transmission using wireless technology. Wireless transmission is the counterpart to wired transmission; compared to wired transmission, it eliminates the cumbersome wiring process and offers greater scalability and lower cost. With the rapid development of wireless technology, its applications are increasingly being accepted across various industries. Combining vehicle tracking with wireless transmission technology has become a hot research topic. This invention proposes a research method for wireless transmission in a distributed Internet of Things (IoT) system based on reinforcement learning. Summary of the Invention

[0004] Technical Problem: The technical problem this invention aims to solve is to propose a research method for wireless transmission in a distributed Internet of Things (IoT) system based on reinforcement learning. Age of Information (AoI) and energy consumption are used as metrics. Each device is independent, monitoring vehicle location information and transmitting it to the base station. Due to limited wireless resources, only some devices can transmit; therefore, the reinforcement learning IPPO algorithm is used to decide which devices can transmit.

[0005] Technical solution: The technical solution adopted in this invention is as follows:

[0006] (1) First, establish the mathematical model corresponding to the wireless transmission problem of the distributed Internet of Things system.

[0007] (2) Build the corresponding simulation environment based on the device transmission mathematical model, and initialize the various parameters used in the environment. These include the AoI of device s, the channel gain set of device s and the base station, the maximum energy that device s can store, the maximum frequency that the device can sample, the energy consumed by device s for sampling, the energy consumed by device s for uplink transmission, and the energy received by device s for downlink transmission.

[0008] (3) On the equipment side, the problem under study is first transformed into a Markov decision process (MDP) based on its characteristics: (S,A,R), where S represents the state space, A represents the action space, and R represents the reward function.

[0009] (4) Finally, the reinforcement learning IPPO algorithm is used to train the agent to make decisions on the transmission scheme that satisfies all constraints and minimizes AoI and energy consumption.

[0010] Preferably, step (1) includes the following steps:

[0011] (11) The distributed Internet of Things system includes N single-antenna Internet of Things devices and a base station with M antennas, wherein each Internet of Things device needs to transmit real-time data packets to the base station; the Internet of Things devices are distributed, that is, the Internet of Things devices cannot communicate with each other, and the base station only knows the wireless channel information of the current time slot, and only knows the statistical information of the channel of the subsequent time slots.

[0012] (12) Introduce AoI to measure the timeliness of information transmission; for base stations, AoI is defined as the number of time slots elapsed since the last successful reception of the latest data packet; for IoT devices, AoI is defined as the number of time slots elapsed since the last successful sampling; set a maximum value M. a To limit the upper limit of AoI, when AoI reaches the maximum value M a After that, the information can be considered meaningless;

[0013] (13) The base station has a fixed energy source, and each IoT device has a radio frequency energy harvesting circuit to receive the energy from the radio frequency signals emitted by the base station and store it in its own battery. The maximum energy storage capacity is B. max The decision to accept the charge is made at the device, which can then communicate the decision to the base station via a control channel. The base station will transmit the charge as long as at least one device needs charging.

[0014] (14) Regarding sampling of IoT devices, device s can determine the interval for sampling the corresponding physical process at time t, and the maximum sampling interval is Δ s，t When the time δ since the most recent sampling s，t Less than Δ s，t When the Nyquist-Shannon sampling theory is satisfied, the sampled information can accurately represent the changes in the physical process; conversely, it cannot accurately represent the changes in the physical process. Using s s，t =1 indicates sampling, denoted by s s，t =0 indicates no sampling, and the energy consumed by sampling is E. s，1In terms of wireless channel transmission, orthogonal frequency division multiple access (OFDMA) is used for the transmission of sampled information, using u s，t =1 indicates uplink transmission. In this case, the device transmits update data packets to the base station, which consumes power E. s，2 , use u s，t =0 indicates downlink transmission. In this case, the base station broadcasts a radio frequency signal to the device for charging, and the charging energy is E. s，3 , use u s，t =2 indicates that device s is idle;

[0015] (15) The optimization objective of the mathematical model is to minimize the weighted average of the AoI and energy consumption on the base station side. The AoI on the base station side is related to the AoI on the device side. If an uplink transmission from the device to the base station occurs at a certain moment, the AoI on the base station side becomes the transmission delay from the device to the base station; otherwise, it is incremented by 1. The energy consumption of the base station is related to whether the device needs to be charged by the downlink transmission from the base station side to the device side. As long as one device needs to be charged, the base station needs to transmit energy at that moment, which leads to energy consumption.

[0016] Preferably, step (3) includes the following steps:

[0017] (31) First, assign an agent to each device, numbered as {1, 2, ..., N}, where N represents the total number of devices;

[0018] (32) Setting of state space S: For agent s, its state space S s Defined as: S s =(A s G s E s F s E s，1 E s，2 E s，3 ), where A s Indicates the AoI, G of device s s F represents the set of uplink and downlink channel gains of device s and base station. s E represents the maximum frequency that device s can sample. s，1 E represents the energy consumed by device s in sampling. s，2 E represents the energy consumed by device s during uplink transmission. s，3 This represents the energy received by device s during downlink transmission; therefore, the total state space S is represented as: S = {S 1 S 2 S N};

[0019] (33) Setting of action space A: For agent s, its action space A sDefined as: A s ={0,1} 2 Where 2 represents the two actions of sampling and transmission; the above action space essentially represents the set of specific schemes for whether the device performs transmission or sampling, and can also be represented as a row vector: A s = {x1, x2}; where if x1 = 1, it means that device s has sampled at this moment; if x2 = 1, it means that device s has performed uplink transmission at this moment; each agent is responsible for deciding whether to sample and transmit, so the total action space is represented as the action space A of all devices. s Cartesian product: A = A 1 ×A 2 ×...×A N ;

[0020] (34) Setting of reward function R: When the device's AoI is small, sampling is performed and a negative reward is given. When the device's AoI is large, uplink transmission is performed and a positive reward is given. When the number of devices transmitting exceeds a value M at a certain moment, a negative reward is given.

[0021] Preferably, step (4) includes the following steps:

[0022] (41) First, initialize the policy network parameters of each agent in the IPPO algorithm. and value network parameters

[0023] (42) Each agent observes the current environmental state and, according to the current policy, moves from action space A. s Choose one action a s ;

[0024] (43) The actions of all agents constitute a joint action: {a 1 ,a 2 ,...,a S};

[0025] (44) The combined action acts on the current state and updates the current state from s to s. ′ At the same time, you will receive an instant reward R;

[0026] (45) After that, the reward will be given as feedback to all agents, updating their policy neural network parameters and value neural network parameters.

[0027] (46) Repeat steps (42-45) until the reward R converges.

[0028] Beneficial Effects: This invention proposes a research method for wireless transmission in a distributed Internet of Things (IoT) system based on reinforcement learning. It uses age of information (AoI) and energy consumption as metrics. Each device is independent and monitors the location information of vehicles and transmits it to the base station. However, due to limited wireless resources, only some devices can transmit. Therefore, the reinforcement learning IPPO algorithm is used to decide which devices can transmit. The IPPO algorithm reuses the sampled data multiple times, solving the problem of low sample utilization, reducing computational time complexity, and improving the efficiency of device transmission. Attached Figure Description

[0029] Figure 1 This is a schematic diagram for training a reinforcement learning algorithm. Detailed Implementation

[0030] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0031] This invention proposes a research method for wireless transmission in distributed Internet of Things (IoT) systems based on reinforcement learning.

[0032] The following is an implementation example:

[0033] (1) First, establish the mathematical model corresponding to the wireless transmission problem of the distributed Internet of Things system.

[0034] (2) Build the corresponding simulation environment based on the device transmission mathematical model, and initialize the various parameters used in the environment. These include the AoI of device s, the channel gain set of device s and the base station, the maximum energy that device s can store, the maximum frequency that the device can sample, the energy consumed by device s for sampling, the energy consumed by device s for uplink transmission, and the energy received by device s for downlink transmission.

[0035] (3) On the equipment side, the problem under study is first transformed into a Markov decision process (MDP) based on its characteristics: (S,A,R), where S represents the state space, A represents the action space, and R represents the reward function.

[0036] (4) Finally, the reinforcement learning IPPO algorithm is used to train the agent to make decisions on the transmission scheme that satisfies all constraints and minimizes AoI and energy consumption.

[0037] The specific steps (1) are as follows:

[0038] (11) Consider a network consisting of N single-antenna IoT devices and a base station with M antennas, where each IoT device needs to transmit real-time data packets to the base station. Assume that the IoT devices are distributed, i.e., the IoT devices cannot communicate with each other, and the base station only knows the wireless channel information of the current time slot, and only knows the statistical information of the channel in the subsequent time slots.

[0039] (12) Introduce AoI to measure the timeliness of information transmission. For base stations, AoI is defined as the number of time slots elapsed since the last successful reception of the latest data packet. For IoT devices, AoI is defined as the number of time slots elapsed since the last successful sampling. A maximum value M can be set. a To limit the upper limit of AoI, when AoI reaches the maximum value M a After that, the information can be considered meaningless.

[0040] (13) Assume the base station has a fixed energy source, and each IoT device has a radio frequency energy harvesting circuit to receive the energy from the radio frequency signal emitted by the base station and store it in its own battery. The maximum energy that can be stored is B. ma The decision to accept a charge is made at the device level, which then transmits the decision to the base station via a control channel. The base station transmits a charge whenever at least one device needs charging.

[0041] (14) Regarding sampling of IoT devices, it is assumed that device s can determine the interval of sampling the corresponding physical process at time t, and the maximum sampling interval is Δ s，t When the time δ since the most recent sampling s，t Less than Δ s，t When the Nyquist-Shannon sampling theory is satisfied, the sampled information can accurately represent the changes in the physical process; otherwise, it cannot accurately represent the changes in the physical process. Using s s，t =1 indicates sampling, denoted by s s，t =0 indicates no sampling, and the energy consumed by sampling is E. s，1 In terms of wireless channel transmission, Orthogonal Frequency Division Multiple Access (OFDMA) is used for the transmission of sampling information, and u... s，t =1 indicates uplink transmission. In this case, the device transmits update data packets to the base station, which consumes power E. s，2 , use u s，t =0 indicates downlink transmission. In this case, the base station broadcasts a radio frequency signal to the device for charging, and the charging energy is E. s，3 , use u s，t =2 indicates that device s is idle.

[0042] (15) The optimization objective of the mathematical model is to minimize the weighted average of the AoI and energy consumption on the base station side. The AoI on the base station side is related to the AoI on the device side. If an uplink transmission from the device to the base station occurs at a certain moment, the AoI on the base station side becomes the transmission delay from the device to the base station; otherwise, it is incremented by 1. The energy consumption of the base station is related to whether the device needs to be charged by the downlink transmission from the base station side to the device side. As long as one device needs to be charged, the base station needs to transmit energy at that moment, which leads to energy consumption.

[0043] The specific steps (3) are as follows:

[0044] (31) First, assign an agent to each device, numbered as {1, 2, ..., N}, where N represents the total number of devices.

[0045] (32) Setting of state space S: For agent s, its state space S s Defined as: S s =(A s G s E s F s E s，1 E s，2 E s，3 ), where A s Indicates the AoI, G of device s s E represents the set of channel gains between device s and the base station. s F represents the maximum energy that device s can store. s E represents the maximum frequency that device s can sample. s，1 E represents the energy consumed by device s in sampling. s，2 E represents the energy consumed by device s during uplink transmission. s，3 Let S represent the energy received by device s during downlink transmission. Therefore, the total state space S is represented as: S = {S...} 1 S 2 S s}

[0046] (33) Setting of action space A: For agent s, its action space A s Defined as: A s ={0,1} 2 Here, 2 represents the two actions of sampling and transmission. The aforementioned action space essentially represents the set of specific schemes for whether the device performs transmission or sampling, and can also be represented as a row vector: A s= {x1, x2}. Where x1 = 1 indicates that device s sampled at this moment, and x2 = 1 indicates that device s performed an uplink transmission at this moment. Each agent is responsible for deciding whether to sample and transmit; therefore, the total action space can be represented as the action space A of all devices. s Cartesian product: A = A 1 ×A 2 ×...×A N .

[0047] (34) Setting of reward function R: When the device's AoI is small, sampling is performed and a negative reward is given. When the device's AoI is large, uplink transmission is performed and a positive reward is given. When the number of devices transmitting exceeds a value M at a certain moment, a negative reward is given.

[0048] The specific steps (4) are as follows:

[0049] (41) First, initialize the policy network parameters of each agent in the IPPO algorithm. and value network parameters

[0050] (42) Each agent observes the current environmental state and, according to the current policy, moves from action space A. s Choose one action a s .

[0051] (43) The actions of all agents constitute a joint action: {a 1 a 2 , ..., a S}

[0052] (44) The combined action acts on the current state and updates the current state from s to s′, while receiving an immediate reward R.

[0053] (45) After that, the reward will be given as feedback to all agents, updating their policy neural network parameters and value neural network parameters.

[0054] (46) Repeat steps (42-45) until the reward R converges. The entire process is as follows: Figure 1 As shown.

Claims

1. A wireless transmission method for a distributed Internet of Things (IoT) system based on reinforcement learning, characterized in that, Includes the following steps: Establish a mathematical model for the wireless transmission problem in a distributed Internet of Things (IoT) system; Based on the mathematical model, a corresponding simulation environment is built, and various parameters used in the environment are initialized, including the AoI of device s in the distributed Internet of Things system, the channel gain set of device s and base station, the maximum energy that device s can store, the maximum sampling frequency that device s can sample, the energy consumed by device s sampling, the energy consumed by device s uplink transmission, and the energy received by device s downlink transmission. On the device side, an intelligent agent is assigned to each device, and the mathematical model is transformed into a Markov decision process: (S,A,R), where S represents the state space, A represents the action space, and R represents the reward function. The agent is trained using the reinforcement learning IPPO algorithm to make decisions on the transmission scheme that satisfies various constraints and minimizes AoI and energy consumption. The mathematical model includes the following: The distributed Internet of Things (IoT) system includes N single-antenna IoT devices and a base station with M antennas. Each IoT device needs to transmit real-time data packets to the base station. The IoT devices are distributed, meaning that the IoT devices cannot communicate with each other. The base station only knows the wireless channel information of the current time slot and only knows statistical information about the channel of the subsequent time slots. AoI (Aspect-Oriented Intelligence) is introduced to measure the timeliness of information transmission. For base stations, AoI is defined as the number of time slots elapsed since the last successful reception of the latest data packet. For IoT devices, AoI is defined as the number of time slots elapsed since the last successful sampling. A maximum value M is set. a To limit the upper limit of AoI, when AoI reaches the maximum value M a After that, the information can be considered meaningless; The base station has a fixed power source, and each IoT device has a radio frequency energy harvesting circuit to receive the energy from the radio frequency signals emitted by the base station and store it in its own battery. The maximum energy storage capacity is B. max Joules, the decision to accept is made at the device, and the device tells the base station the decision through the control channel; The base station will transmit as long as at least one device needs charging; In terms of IoT device sampling, device s determines the sampling interval for the corresponding physical process at time t, and the maximum sampling interval is Δ. s,t When the time δ since the most recent sampling s,t Less than Δ s,t When the Nyquist-Shannon sampling theory is satisfied, the sampled information accurately represents the changes in the physical process; conversely, it cannot accurately represent the changes in the physical process. Using s... s,t =1 indicates sampling, denoted by s s,t =0 indicates no sampling, and the energy consumed by sampling is E. s,1 In terms of wireless channel transmission, orthogonal frequency division multiple access (OFDMA) is used for the transmission of sampled information, using u s,t =1 indicates uplink transmission. In this case, the device transmits update data packets to the base station, which consumes power E. s,2 , use u s,t =0 indicates downlink transmission. In this case, the base station broadcasts a radio frequency signal to the device for charging, and the charging energy is E. s,3 , use u s,t =2 indicates that device s is idle.

2. The wireless transmission method for a distributed Internet of Things system based on reinforcement learning as described in claim 1, characterized in that, On the device side, the mathematical model is transformed into a Markov decision process: (S, A, R), including the following: Each intelligent agent is numbered as {1,2,...,N}, where N represents the total number of devices; State space S configuration: For agent s, its state space S s Defined as: S s =(A s G s E s ,F s E s,1 E s,2 E s,3 ), where A s Indicate the AoI, G of device s s F represents the set of uplink and downlink channel gains of device s and base station. s E represents the maximum frequency that device s can sample. s,1 E represents the energy consumed by device s in sampling. s,2 E represents the energy consumed by device s during uplink transmission. s,3 This represents the energy received by device s during downlink transmission; therefore, the total state space S is represented as: S = {S 1 ,S 2 ,...,S N }; Setting of action space A: For agent s, its action space A s Defined as: A s ={0,1} 2 Where 2 represents the two actions of sampling and transmission; the action space essentially represents the set of specific schemes for whether the device performs transmission or sampling, and can also be represented as a row vector: A s = {x1, x2}; where if x1 = 1, it means that device s has sampled at this moment; if x2 = 1, it means that device s has performed uplink transmission at this moment; each agent is responsible for deciding whether to sample and transmit, so the total action space is represented as the action space A of all devices. s Cartesian product: A = A 1 ×A 2 ×...×A N ; The reward function R is set as follows: when the device's AoI is small, sampling is performed and a negative reward is given; when the device's AoI is large, uplink transmission is performed and a positive reward is given; when the number of devices transmitting at a certain moment exceeds a value M, a negative reward is given.

3. The wireless transmission method for a distributed Internet of Things system based on reinforcement learning as described in claim 1, characterized in that, The agent is trained using the reinforcement learning IPPO algorithm, which includes the following steps: Initialize the policy network parameters of each agent in the IPPO algorithm and value network parameters Each agent observes the current environmental state and, based on the current policy, moves from action space A... s Choose one action a s ; The actions of all agents constitute a joint action: {a 1 ,a 2 ,...,a S }; The combined action acts on the current state, updates the current state from s to s′, and simultaneously receives an immediate reward R; The reward will be given as feedback to all agents, updating their policy neural network parameters and value neural network parameters; Repeat the above steps until the reward R converges.

4. The wireless transmission method for a distributed Internet of Things system based on reinforcement learning as described in claim 1, characterized in that, The optimization objective of the mathematical model is to minimize the weighted average of AoI and energy consumption on the base station side.

5. The wireless transmission method for a distributed Internet of Things system based on reinforcement learning as described in claim 4, characterized in that, The AoI on the base station side is related to the AoI on the device side. If an uplink transmission from the device to the base station occurs at a certain moment, the AoI on the base station side becomes the transmission delay from the device to the base station; otherwise, it is incremented by 1. The energy consumption of the base station is related to whether the device needs to be charged by downlink transmission from the base station side to the device side. As long as one device needs to be charged, the base station needs to transmit energy at that moment, which leads to energy consumption.

Images