Data processing method, device, equipment, and storage medium

By using trained reinforcement learning models in Wi-Fi communication systems to optimize the physical layer rate, the problems of low throughput rates and high power consumption in the prior art are solved, and more efficient data transmission and lower power consumption are achieved.

CN115412437BActive Publication Date: 2024-11-12GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210987455.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-17
Publication Date
2024-11-12
Estimated Expiration
2042-08-17

AI Technical Summary

Technical Problem

In the existing Wi-Fi communication systems, physical layer rate adjustment based on sampling technology cannot effectively improve the throughput rate of data transmission, resulting in low bandwidth utilization and high power consumption.

Method used

The trained reinforcement learning model is adopted to determine appropriate physical layer rate parameters based on the measurement data of wireless channel quality, and optimize data transmission through reinforcement learning algorithms to reduce bandwidth losses caused by improper rates and improve channel capacity utilization.

Benefits of technology

The throughput rate of data transmission is improved, the power consumption of chip is reduced, and the efficiency and power consumption performance of data packet transmission are improved by maximizing channel capacity utilization and reducing unnecessary retransmission.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115412437B_ABST
    Figure CN115412437B_ABST
Patent Text Reader

Abstract

The present application provides a data processing method and apparatus, a device, and a storage medium; wherein the data processing method comprises: obtaining a first state, wherein the first state comprises measurement data for reflecting the quality of a wireless channel; based on a trained reinforcement learning model, determining a first target action corresponding to the first state, wherein the first target action comprises a parameter value of a parameter affecting a physical layer rate; wherein the first target action is used to send first data to a first device; the trained reinforcement learning model is obtained by training a first reward based on a second state and a second target action corresponding to the second state, wherein the first reward is obtained based at least on a first throughput rate achieved when the second target action sends the second data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to communication technology, and to but is not limited to data processing methods and devices, equipment, and storage media. Background Art

[0002] In the Wireless Fidelity (Wi-Fi) communication system, the transmitter often uses sampling technology to dynamically adjust the physical layer rate (Wi-Fi Data Rate) of Wi-Fi, that is, every few transmissions, it will try to send data packets at a higher physical layer rate, and then collect feedback indicators such as the failure rate and number of retransmissions of data packets. If these indicators are good, it means that the channel conditions allow data packets to be sent and received at a higher physical layer rate, so this higher physical layer rate is used to send data packets. Conversely, if these indicators are poor, it means that the current channel conditions do not allow data packets to be sent and received at a higher physical layer rate, and the current rate is kept unchanged. On the other hand, if the indicators of sending and receiving data packets at the current rate deteriorate, it means that the channel conditions no longer allow data packets to be sent and received at the current rate, and the transmitter needs to lower the physical layer rate.

[0003] However, the physical layer rate adjustment implemented based on the above method cannot obtain a better throughput rate. Summary of the invention

[0004] In view of this, the data processing method, device, equipment, and storage medium provided in the present application can improve the throughput rate of data transmission.

[0005] According to one aspect of an embodiment of the present application, a data processing method is provided, including: acquiring a first state, the first state including measurement data for reflecting the quality of a wireless channel; determining a first target action corresponding to the first state based on a trained reinforcement learning model, the first target action including a parameter value of a parameter affecting a physical layer rate; wherein the first target action is used to send first data to a first device; the trained reinforcement learning model is trained based on a second state and a first reward for a second target action corresponding to the second state, the first reward being obtained based at least on a first throughput rate achieved when the second target action sends second data.

[0006] According to another aspect of an embodiment of the present application, a data processing device is provided, including: a first acquisition module, configured to acquire a first state, the first state including measurement data for reflecting the quality of a wireless channel; a first determination module, configured to determine a first target action corresponding to the first state based on a trained reinforcement learning model, the first target action including a parameter value of a parameter affecting the physical layer rate; wherein the first target action is used to send first data to a first device, the trained reinforcement learning model is obtained by training a first reward for a second target action corresponding to a second state and the second state, and the first reward is obtained based at least on a first throughput rate achieved when the second target action sends the second data.

[0007] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0008] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0009] In the embodiment of the present application, since the trained reinforcement learning model is not an empirical formula obtained by manual adjustment, but is obtained based on the reinforcement learning algorithm, it can discover and summarize the deeper laws in the state space. Therefore, compared with the empirical formula, the target action (i.e., the physical layer rate) output by it is more in line with the actual transmission environment; thus, on the one hand, the bandwidth loss caused by sending data at an inappropriate physical layer rate is reduced, thereby improving the actual throughput capacity; this is because: the physical layer rate that is inconsistent with the actual capacity of the channel will cause the throughput rate to deteriorate; if the physical layer rate is lower than the actual capacity of the channel, then it is obvious that the utilization rate of the channel capacity cannot be maximized; if the physical layer rate is higher than the actual capacity of the channel, a large number of error packets and retransmissions will be generated, and unnecessary retransmissions will waste the available capacity of the channel. The data processing method based on the empirical formula inevitably brings greater bandwidth loss.

[0010] On the other hand, the improvement of actual throughput can also improve the power consumption performance of the chip. The reason is that if the channel capacity can be maximized or used to the maximum extent, the data transmitter can complete the data packet reception at a faster speed, allowing the chip to enter the low power mode more time. Similarly, by avoiding unnecessary data retransmission, the chip can also complete the data transmission as soon as possible, thus having more time to enter the low power mode.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0014] Figure 1 A schematic diagram of the implementation flow of the model training method provided in the embodiment of the present application;

[0015] Figure 2 This is a schematic diagram of the Actor-Critic algorithm principle;

[0016] Figure 3 A schematic diagram of the implementation flow of the data processing method provided in the embodiment of the present application;

[0017] Figure 4 This is a schematic diagram of the principle of reinforcement learning algorithm;

[0018] Figure 5 Schematic diagram of the structure of the optimization strategy estimator and value estimator;

[0019] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0020] Figure 7 A schematic diagram of the hardware entity of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0023] In the following description, reference is made to “some embodiments”, “this embodiment”, “embodiments of the present application” and examples, etc., which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0024] It should be noted that the terms "first\second\third\fourth\fifth" etc. involved in the embodiments of the present application do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third\fourth\fifth" etc. can be interchanged where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0025] In the embodiment of the present application, the second device uses the trained reinforcement learning model to calculate the physical layer rate used for data transmission, and sends the first data to the first device based on at least the calculation result. That is, the second device obtains the first state, and determines the first target action corresponding to the first state based on the trained reinforcement learning model, and the first target action is used to send the first data to the first device;

[0026] The first state includes measurement data for reflecting the quality of the wireless channel, the first target action includes parameter values ​​of parameters affecting the physical layer rate, the trained reinforcement learning model is trained based on the second state and the first reward of the second target action corresponding to the second state, and the first reward is obtained based at least on the first throughput rate achieved when the second target action sends the second data.

[0027] In order to facilitate the understanding of the data processing method and device provided in the embodiments of the present application, the training process of the trained reinforcement learning model is first described, and then the implementation process of the data processing method is described.

[0028] Figure 1 A schematic diagram of the implementation flow of the model training method provided in the embodiment of the present application, such as Figure 1 As shown, it includes the following steps 101 to 105:

[0029] Step 101: The fourth device acquires a second state, where the second state includes measurement data for reflecting the quality of a wireless channel;

[0030] Step 102: The fourth device determines a second target action corresponding to the second state based on the current reinforcement learning model, where the second target action includes a parameter value of at least one parameter affecting the physical layer rate;

[0031] Step 103: The fourth device sends second data to the third device based at least on the second target action;

[0032] Step 104: After sending the second data, the fourth device measures the corresponding first throughput rate;

[0033] Step 105: The fourth device uses the first throughput rate as a first reward to update parameter values ​​of model parameters of the reinforcement learning model.

[0034] The fourth device iterates the above steps repeatedly until the iteration cutoff condition is met, thereby obtaining a trained reinforcement learning model.

[0035] Understandably, manually derived empirical formulas generally have limited expressive power. Judgments on output values ​​with relatively large state spaces are relatively trivial and difficult to summarize. The rules that can be summarized are often relatively obvious, and deeper rules are difficult to discover and summarize.

[0036] In view of this, in an embodiment of the present application, the fourth device uses a reinforcement learning method to perform machine learning on the reinforcement learning model rather than summarizing it based on artificial experience, so as to dig out deeper transmission rules in the actual transmission environment, and then obtain a reinforcement learning model that is more in line with the actual transmission environment than the artificially derived empirical formula; therefore, the trained reinforcement learning model is very beneficial for the second device to obtain a better physical layer rate, so that the second device can achieve a better throughput rate when sending data based on the physical layer rate predicted by the model.

[0037] It should be noted that the executor of the model training method provided in the embodiment of the present application may also be a second device, that is, offline model training and online model reasoning can be implemented in the same device.

[0038] The following describes further optional implementations and related terms of each of the above steps.

[0039] In step 101, the fourth device obtains a second state, where the second state includes measurement data reflecting the quality of a wireless channel.

[0040] In an embodiment of the present application, the measurement data includes at least the state data at the current moment. In other embodiments, the measurement data includes the state data at the current moment and the state data at the previous moment. These state data can be statistical data based on the hardware of the fourth device or the receiving end device (i.e., the third device), including but not limited to basic data from RF, PHY and MAC layer statistics.

[0041] For example, in some embodiments, the measurement data includes at least one of the following: received signal strength indicator (RSSI), error vector magnitude (EVM), signal to noise ratio (SNR), retry ratio, and packet loss rate (PER). Of course, in the embodiments of the present application, it is not limited to the parameter values ​​of these parameters. In short, the parameter values ​​of the parameters that can reflect the quality of the wireless channel are sufficient.

[0042] In step 102, the fourth device determines a second target action corresponding to the second state based on the current reinforcement learning model, where the second target action includes a parameter value of at least one parameter that affects the physical layer rate.

[0043] In some embodiments, the second target action includes at least parameter values ​​of the following parameters: Modulation and Coding Scheme (MCS) index, bandwidth, long / short guard interval (guard interval), spatial stream number (SSN). Of course, in the embodiments of the present application, it is not limited to the parameter values ​​of the above parameters, and the second target action may also include parameter values ​​of other parameters that affect the physical layer rate.

[0044] Due to the complexity of wireless channels and the contingency of sudden factors in actual application scenarios, direct modeling often cannot effectively express channel characteristics and interaction features. Therefore, in the embodiments of the present application, no limitation is made to the reinforcement learning algorithm, that is, no limitation is made to the reinforcement learning model and training method. In some embodiments, the fourth device may adopt a model-free (Model-Free RL) reinforcement learning algorithm, which mainly includes two major types of algorithms: policy algorithms (Policy Optimization) and Q-Learning. Specifically, it includes a cluster of implementation algorithms, including but not limited to DQN (Deep Q Networks), Policy Gradient (Policy Gradient), A2C (Advantage Actor-Critic), A3C (Asynchronous Advantage Actor-Critic) and Deep Deterministic Policy Gradient Algorithm (Deep Deterministic Policy Gradient, DDPG).

[0045] In step 103, the fourth device sends second data to the third device based at least on the second target action.

[0046] The second data is sent to the third device at least based on the second target action. In some embodiments, after obtaining the second target action, the fourth device may not extend the action, and directly send the second data to the third device based on the action; further, the fourth device may extend at least one fourth target action based on the second target action and a pre-configured physical layer rate list when sending the second data fails based on the second target action, and try to send the second data in sequence based on the second target action;

[0047] In other embodiments, sending the second data to the third device at least based on the second target action includes: the fourth device first expands at least one fourth target action based on the second target action and a preconfigured physical layer rate list, and then tries to send the second data to the first device in sequence based on the second target action and the at least one fourth target action.

[0048] Further, in some embodiments, the at least one fourth target action includes multiple fourth target actions, and the fourth device can sort the multiple fourth target actions according to the SNR requirements; the fourth device attempts to send the second data in turn according to the sorting results.

[0049] For example, according to the SNR requirement, three actions are obtained, namely: an action with a throughput rate lower than the second target action, an action with suboptimal reliability, and an action with optimal reliability. When the fourth device fails to send the second data based on the second target action, the fourth device attempts to send the second data to the third device in the order of the three actions.

[0050] In some embodiments, the fourth device encodes and modulates the second data based on the second target action, thereby sending the second data to the third device. That is, the second data is sent to the third device at the physical layer rate corresponding to the second target action.

[0051] In step 104, after sending the second data, the fourth device measures a corresponding first throughput.

[0052] In step 105, the fourth device uses the first throughput rate as a first reward to update parameter values ​​of model parameters of the current reinforcement learning model.

[0053] Considering that in some communication systems, the throughput rate is often greater than or equal to 0, and the reinforcement learning algorithm usually requires a negative reward to adjust the increase or decrease of the parameter value of the model parameter of the reinforcement learning model. If the rewards are all positive, then all actions will be rewarded, so the parameter value of the model parameter of the reinforcement learning model is saturated before it reaches the optimal value.

[0054] In view of this, in some embodiments, the fourth device can correct the first reward through the value network to obtain a corrected first reward; input the corrected first reward into the loss function, and update the model parameters of the current reinforcement learning model according to the gradient of the loss function; in this way, it is avoided that the parameter values ​​of the model parameters of the reinforcement learning model cannot be trained to meet the requirements because the rewards are all positive values.

[0055] In some embodiments, the fourth device may use an RL baseline improvement algorithm, that is, to modify the reward. For example, the Actor-Critic algorithm in the RL algorithm may be used to modify the reward by the valuation of the value function, and the modified value is then used to optimize the strategy estimator and the value estimator, which may be two deep neural networks. Figure 2As shown in the figure, Policy is an example of a reinforcement learning model, Value Function is an example of a value network, and Environment is the environment (which can be understood as the actual environment where the fourth device is located). Policy plays the role of Actor, and Value Function plays the role of Critic. The input of Actor is a state, a corrected reward (TD error), that is, the corrected first throughput rate. Actor, as a policy network, adjusts the probability of taking various actions under different states based on the corrected reward.

[0056] Critic is a value-based learning method, so it can be updated in a single step and calculate the reward and punishment value (i.e., reward) for each step. Then the two are combined, the Actor chooses the action, and the Critic tells the Actor whether the action it chooses is appropriate. In this process, the Actor iterates continuously to obtain the reasonable probability of selecting each action in each state, and the Critic also iterates continuously to continuously improve the reward and punishment value for selecting each action in each state; among them, the action is the candidate action, and the target action comes from one of the multiple candidate actions.

[0057] The fourth device iterates the above steps repeatedly until the iteration cutoff condition is met, thereby obtaining a trained reinforcement learning model. It can be understood that the state input into the latest reinforcement learning model is different in each iteration.

[0058] In the embodiment of the present application, there is no limitation on the iteration cutoff condition, which may be that the number of iterations reaches a preset number, or that the first reward tends to converge.

[0059] In the embodiment of the present application, there is no limitation on what reinforcement learning algorithm and what structure of reinforcement learning model the fourth device adopts. For example, Q learning, Sarsa, Deep QNetwork, PolicyGradients, Model based RL, Actor-Critic and other reinforcement learning algorithms can be used to implement the training of the reinforcement learning model. In short, based on the state signal and action, the reward given by the environment is obtained; based on the reward, the model parameters of the reinforcement learning model are updated to obtain the final trained reinforcement learning model, which can then be used in the second device, that is, based on the currently measured state data, a suitable target action can be obtained, and based on the target action, the data to be sent is sent to the first device.

[0060] After obtaining the trained reinforcement learning model, since the trained reinforcement learning model meets the performance index requirements of data transmission, such as the index requirements of throughput rate, the model can be applied to the data processing method of the second device. Figure 3 A schematic diagram of the implementation flow of the data processing method provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes the following steps 301 to 303:

[0061] Step 301: A second device obtains a first state, where the first state includes measurement data reflecting quality of a wireless channel.

[0062] The parameters included in the first state are the same as the parameters included in the second state mentioned above, except that the two are information obtained at different times, and the parameter values ​​of the same parameter may be different. These measurement data can be statistical data based on the hardware of the second device or the receiving device (i.e., the first device), including but not limited to basic data from RF, PHY and MAC layer statistics.

[0063] For example, in some embodiments, the first state includes at least one of the following measurement data: RSSI, EVM, SNR, retransmission rate, packet loss rate. Of course, in the embodiments of the present application, it is not limited to the parameter values ​​of these parameters. In short, the parameter values ​​of the parameters that can reflect the quality of the wireless channel are sufficient.

[0064] Step 302: The second device determines a first target action corresponding to the first state based on the trained reinforcement learning model, where the first target action includes a parameter value of a parameter affecting a physical layer rate;

[0065] Step 303: The second device sends first data to the first device based at least on the first target action.

[0066] In the embodiment of the present application, since the trained reinforcement learning model is not an empirical formula obtained by manual adjustment, but is obtained based on the reinforcement learning algorithm, it can discover and summarize the deeper laws in the state space. Therefore, compared with the empirical formula, the target action (i.e., the physical layer rate) output by it is more in line with the actual transmission environment; thus, on the one hand, the bandwidth loss caused by sending data at an inappropriate physical layer rate is reduced, thereby improving the actual throughput capacity; this is because: the physical layer rate that is inconsistent with the actual capacity of the channel will cause the throughput rate to deteriorate; if the physical layer rate is lower than the actual capacity of the channel, then it is obvious that the utilization rate of the channel capacity cannot be maximized; if the physical layer rate is higher than the actual capacity of the channel, a large number of error packets and retransmissions will be generated, and unnecessary retransmissions will waste the available capacity of the channel. The data processing method based on the empirical formula inevitably brings greater bandwidth loss.

[0067] On the other hand, the improvement of actual throughput can also improve the power consumption performance of the chip. The reason is that if the channel capacity can be maximized or used to the maximum extent, the data transmitter can complete the data packet reception at a faster speed, allowing the chip to enter the low power mode more time. Similarly, by avoiding unnecessary data retransmission, the chip can also complete the data transmission as soon as possible, so that it has more time to enter the low power mode.

[0068] It should be noted that the second device sends the first data to the first device at least based on the first target action. In some embodiments, the first data may be sent to the first device based on the first target action. Further, in some embodiments, if the second device fails to send the first data based on the first target action, the second device expands to obtain at least one third target action based on the first target action and a pre-configured physical layer rate list, and then tries to send the first data to the first device in turn based on the at least one third target action.

[0069] In other embodiments, the sending of the first data to the first device at least based on the first target action may also be based on the first target action and a preconfigured physical layer rate list, expanding to obtain at least one third target action, and then based on the first target action and the at least one third target action, sequentially attempting to send the first data to the first device.

[0070] It is understandable that when the second device sends the first data based on the first target action, there may also be a situation where the first data fails to be sent, that is, the second device does not receive the first data. Therefore, the second device first expands to obtain at least one third target action based on the first target action and the pre-configured physical layer rate list; then, based on the first target action and the at least one third target action, sends the first data to the first device; wherein the at least one third target action is used to resend the first data to the first device in the case of a failure in sending the first data.

[0071] Further, in some embodiments, the at least one third target action includes multiple third target actions, and the second device can sort the multiple third target actions according to the SNR requirements; the second device attempts to send the first data in turn according to the sorting results.

[0072] For example, according to the SNR requirement, three actions are obtained, namely: an action with a throughput rate lower than the first target action, an action with suboptimal reliability, and an action with optimal reliability. In the case where the second device fails to send the first data based on the first target action, the second device attempts to send the first data to the first device in the order of the three actions.

[0073] The pre-configured physical layer rate list actually includes the action corresponding to each physical layer rate, such as partial information of the list shown in Table 1 below.

[0074] Table 1 DATA RATES FOR VARIOUS 802.11AC CONFIGURATIONS

[0075]

[0076] In some embodiments, the second device may encode and modulate the first data based on the first target action, thereby sending it to the first device; wherein the first target action includes at least: MCS index, bandwidth (Channel width), guard interval (Guard Interval, GI), spatial streams (Spatial streams) number. Of course, in the embodiments of the present application, it is not limited to the parameter values ​​of the above parameters. The first target action may also include parameter values ​​of other parameters that affect the physical layer rate. Similarly, when the second device sends the first data based on the third target action, it encodes and modulates the first data based on the third target action. Thereby achieving the retransmission of the first data.

[0077] In some embodiments, the trained reinforcement learning model is trained in a general environment, and the trained reinforcement learning model may not be close to the personalized environment of the second device. In view of this, in some embodiments, the data processing method also includes: after the second device sends the first data, it measures the corresponding second throughput rate; using the second throughput rate as a second reward, updating the parameter values ​​of the model parameters of the trained reinforcement learning model; in this way, based on the throughput rate output by the actual environment of the second device, the parameter values ​​of the model parameters of the trained reinforcement learning model are updated, so that the updated reinforcement learning model is more in line with the actual environment of the second device, thereby obtaining a reinforcement learning model that meets the user's customization, so that in subsequent use, it can predict the target action that is more in line with the actual environment of the second device, thereby further improving the throughput rate.

[0078] Furthermore, in some embodiments, the second device modifies the second reward through the value network to obtain a modified second reward; inputs the modified second reward into the loss function to obtain the gradient of the loss function; and updates the model parameters of the trained reinforcement learning model according to the gradient of the loss function.

[0079] The correction algorithm for the second reward is the same as the correction algorithm for the first reward mentioned in the previous training phase, so it will not be repeated here.

[0080] It should be noted that the data processing method provided in the embodiment of the present application can be used for devices in a Wi-Fi communication system, that is, devices in a wireless local area network of the IEEE 802.11 standard. Accordingly, the second device and the first device are Wi-Fi devices, and the third device and the fourth device are also Wi-Fi devices.

[0081] The data processing method provided in the embodiment of the present application can also be applied to other types of wireless communication system equipment, such as the fourth generation mobile communication system (4G), the fifth generation mobile communication technology (5G) new radio (NR) system or future communication systems.

[0082] During the implementation process, the first device, the second device, the third device and the fourth device may be various types of devices with communication capabilities, such as mobile phones, tablet computers, routing devices, televisions, smart wearable devices (such as bracelets, watches), etc.

[0083] In the embodiment of the present application, the second device and the fourth device may be the same device or different devices. The first device and the third device may be the same device or different devices. However, the first device and the second device are different devices, and the third device and the fourth device are different devices.

[0084] The sampling-based Wi-Fi rate adjustment strategy is slow to respond to environmental changes and is inflexible, which is reflected in the adjustment strategy. If the rate is adjusted too aggressively, the actual bandwidth throughput will be unstable and fluctuate greatly. If the rate is adjusted too passively, it will not be able to respond to environmental changes in a timely manner, resulting in a decrease in throughput performance, which is lower than the actual maximum capacity. However, the speed of this adjustment is difficult to grasp and often has no universal significance.

[0085] On the other hand, the empirical formulas obtained by manual adjustment generally have limited expressive power. The output value judgments for larger state spaces are very trivial and difficult to summarize. The rules that can be summarized are often relatively obvious, and deeper rules are difficult to discover and summarize.

[0086] Based on this, an exemplary application of an embodiment of the present application in a practical application scenario will be described below.

[0087] In the embodiment of the present application, based on the reinforcement learning algorithm, various input states on the input Wi-Fi channel can be inferred to directly obtain the optimal action.

[0088] The principle of reinforcement learning algorithm is as follows Figure 4 As shown in Figure 1, in the process of reinforcement learning, the agent and the environment are constantly interacting. The agent here includes the transceiver device mentioned above, mainly the second device or the fourth device, and the environment refers to the transmission environment between the data transceiver devices. The agent obtains the state S in the environment. t , the agent uses this state S t Output an action A t , which is a decision. Then this decision A t Put it into the environment, and the environment will make decisions based on the decision A taken by the agent. t , output the next state S t+1 And the reward R for the current decision t+1 The goal of the agent is to extract as much reward as possible from the environment.

[0089] According to the characteristics of reinforcement learning, the entire training process includes several parts: environment, agent, state, action and reward.

[0090] A. Status (i.e. the status mentioned above): corresponds to Wi-Fi measurement data, based on Wi-Fi hardware statistics, including but not limited to basic data from RF, PHY and MAC layer statistics. Basic data include but are not limited to: RSSI, EVM, SNR, retransmission rate and PER.

[0091] B. Action (i.e. the action mentioned above): The output of the algorithm is a selection of a parameter, the so-called action, including: MCS index, bandwidth, long / short guard interval, SSN, and parameters that may affect Wi-Fi data rate in the future. In order to facilitate the formation of the final action output, all combinations of these parameters will be assigned a unique action identifier as a selection of a set of parameters.

[0092] C. Reward: The actual throughput rate measured.

[0093] D. Due to the complexity of Wi-Fi channels and the randomness of sudden factors in actual application scenarios, direct modeling often cannot effectively express channel characteristics and interaction features. Therefore, a model-free (Model-Free RL) reinforcement learning algorithm is used here, mainly including two major types of algorithms: Policy Optimization and Q-Learning. Specifically, it includes a cluster of implementation algorithms, including but not limited to: DQN, Policy Gradient, A2C, A3C and DDPG.

[0094] E. Due to the characteristics of Wi-Fi rate, the feedback result is usually the actual throughput rate, which is a positive value greater than or equal to zero, and usually the reinforcement learning algorithm requires a negative reward to adjust the increase or decrease of parameters. If they are all positive values, then all actions will be rewarded, which will quickly saturate the action selection module, so the RL baseline (baseline) improvement algorithm can be used, that is, the reward can be corrected. Specifically, the Actor-Critic algorithm in the RL algorithm can be used to correct the reward by the valuation of the value function, and the corrected value is then used to optimize the policy estimator (estimator) and the value estimator. The latter two are two deep neural networks, such as Figure 5 shown.

[0095] F. Since the Wi-Fi data rate often requires a set of results to be input into the hardware for trial one by one, the optimal result output by RL can refer to the Wi-Fi data rate algorithm for extended processing, including but not limited to the following examples: starting from the optimal result, sorting by SNR requirements, four rates are obtained, namely the data rate with the highest throughput, the data rate with the second highest throughput, the data rate with the second best reliability, and the data rate with the best reliability.

[0096] G. The training process of the Actor-Critic algorithm includes the following steps 601 to 605:

[0097] Step 601, sample {s i ,a i} comes from the policy function π θ (a|s); where s is the abbreviation of state. i It refers to the state at the i-th moment or step, and a is the abbreviation of action;

[0098] Step 602, through the value function For the sample {s i ,a i}The rewards obtained are corrected;

[0099] Step 603, calculate

[0100] Step 604, calculate

[0101] Step 605,

[0102] It should be noted that r is the abbreviation of reward, π represents the policy function, θ represents the internal parameter of π,θ (a|s) represents the policy with parameters θ for state s and action a. is the value function, i.e. the function of the Critic, V is the abbreviation of value, which represents the value evaluation under the internal parameter φ of the strategy π. γ is the correction parameter (hyperparameter), Σ is the summation function, and log is the logarithmic function. is the partial derivative function, is the partial derivative of the internal parameters of the policy network. α is the modulation parameter (hyperparameter).

[0103] Among them, state, action and reward have been described above, and the other parameters are standard parameters of the Actor-Critic algorithm. The policy function π is the Actor, and the value function V is the Critic. Its structure is Figure 5 The fully connected network layers are shown.

[0104] The above method can reduce the bandwidth loss caused by improper rate attempts, thereby improving the actual throughput. The reason is that the rate that is inconsistent with the actual capacity of the channel leads to poor throughput. If the sending rate is lower than the actual capacity of the channel, then it is obviously impossible to maximize the utilization of the channel capacity; if the sending rate is higher than the actual capacity of the channel, a large number of error packets and retransmissions will be generated, and unnecessary retransmissions will waste the available capacity of the channel. Traditional sampling-based algorithms inevitably have problems with sampling overhead and bandwidth loss.

[0105] The above method can also improve the power consumption performance of the chip. The reason is: if the channel capacity can be maximized, the sending device can complete the data packet reception at a faster speed, so that the chip has more time to enter the low power mode. Similarly, by avoiding unnecessary retransmissions, the chip can also complete data transmission as soon as possible, so that it has more time to enter the low power mode.

[0106] The data processing method based on reinforcement learning can quickly identify the current channel and interference characteristics, including but not limited to spectrum characteristics, retransmission rate characteristics, and packet error rate characteristics, etc. As described in the previous state, a state vector is formed, and the reinforcement learning algorithm is used to learn the optimal rate that should be output based on the channel feedback reward. After the reinforcement learning network training is completed, the corresponding optimal rate can be directly predicted based on the current state, avoiding the sampling overhead, bandwidth loss, and adjustment lag of the sampling algorithm.

[0107] The above method includes steps A, B, C, D, E, F, and G, where E can be improved / replaced by E'. E' here can be other algorithms that improve learning stability, or it can be another reward indicator (rather than the actual throughput rate), so that a simpler RL algorithm can be used.

[0108] As a utilization of historical state data, the policy neural network and valuation neural network here can also be sequence data networks of RNN or Transformer technology, so that more features can be extracted from historical sequence data, thereby outputting the optimal result that is not limited to the current state, but can also be the optimal result of the recent history.

[0109] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.; or, steps in different embodiments may be combined into a new technical solution.

[0110] Based on the foregoing embodiments, the embodiments of the present application provide a model training device and a data processing device. The two devices include the modules included and the units included in the modules, which can be implemented by a processor; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be an AI acceleration engine (such as NPU, etc.), GPU, central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0111] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the data processing device 600 includes:

[0112] A first acquisition module 601 is configured to acquire a first state, where the first state includes measurement data reflecting the quality of a wireless channel;

[0113] The first determination module 602 is configured to determine a first target action corresponding to the first state based on a trained reinforcement learning model, wherein the first target action includes a parameter value of a parameter affecting the physical layer rate; wherein the first target action is used to send first data to a first device, and the trained reinforcement learning model is trained based on a second state and a first reward for a second target action corresponding to the second state, and the first reward is obtained based at least on a first throughput rate achieved when the second target action sends the second data.

[0114] In some embodiments, the data processing device 600 also includes: a second acquisition module, configured to acquire a second state, the second state including measurement data for reflecting the quality of the wireless channel; a second determination module, configured to determine a second target action corresponding to the second state based on a reinforcement learning model, the second target action including a parameter value of a parameter affecting the physical layer rate; a second sending module, configured to send second data to a third device based on the second target action; a first measurement module, configured to measure the corresponding first throughput rate after sending the second data; an update module, configured to use the first throughput rate as a first reward to update the parameter value of the model parameter of the current reinforcement learning model; an iteration module, configured to trigger the above modules until the iteration cutoff condition is met to obtain a trained reinforcement learning model.

[0115] In some embodiments, the update module is further configured to: after sending the first data, measure the corresponding second throughput rate; and use the second throughput rate as a second reward to update the parameter values ​​of the model parameters of the trained reinforcement learning model.

[0116] In some embodiments, the update module is configured to: modify the reward through the value network to obtain a modified reward; the reward is the first reward or the second reward; input the modified reward into the loss function, and update the model parameters of the reinforcement learning model according to the gradient of the loss function.

[0117] In some embodiments, the data processing device 600 also includes a first sending module, configured to: determine at least one third target action based on the first target action and a preconfigured physical layer rate list; wherein the at least one third target action is used to resend the first data to the first device in the event of a failure in sending the first data.

[0118] In some embodiments, the first sending module is configured to: based on the first target action and the at least one third target action, sequentially attempt to send the first data to the first device.

[0119] In some embodiments, the first state includes at least one of the following: RSSI, EVM, SNR, retransmission rate, and packet loss rate.

[0120] In some embodiments, the first sending module 603 is configured to: based on the first target action, encode and modulate the first data, thereby sending it to the first device; wherein the first target action includes at least: MCS index, bandwidth, protection interval, and spatial stream number.

[0121] In some embodiments, the data processing apparatus 600 is disposed in the second device, and the second device and the first device are Wi-Fi devices.

[0122] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0123] It should be noted that the division of modules by the device described in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0124] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0125] An embodiment of the present application provides an electronic device, Figure 7 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device 700 includes a memory 701 and a processor 702, the memory 701 stores a computer program that can be run on the processor 702, and the processor 702 implements the steps of the model training method provided in the above embodiment when executing the program, or implements the steps of the data processing method provided in the above embodiment when executing the program.

[0126] It should be noted that the memory 701 is configured to store instructions and applications executable by the processor 702, and can also cache data to be processed or processed by the processor 702 and various modules in the electronic device 700 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0127] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0128] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0129] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0130] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0131] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0132] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0134] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0135] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0136] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0137] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0138] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0139] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0140] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0141] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a first state, wherein the first state includes measurement data for reflecting the quality of a wireless channel; wherein the measurement data includes state data at a current moment and state data at a previous moment; and the measurement data includes at least one of the following: received signal strength indication, error vector magnitude, signal-to-noise ratio, retransmission rate, and packet loss rate; Based on the trained reinforcement learning model, determining a first target action corresponding to the first state, the first target action including a parameter value of a parameter affecting a physical layer rate; wherein the first target action is used to send first data to a first device; the trained reinforcement learning model is obtained by training based on a second state and a first reward for a second target action corresponding to the second state, and the first reward is obtained based on at least a first throughput rate achieved when the second target action sends second data; Based on the first target action, the first data is coded and modulated, and then sent to the first device.

2. The method according to claim 1, characterized in that: The training process of the reinforcement learning model includes: Acquire a second state, wherein the second state includes measurement data reflecting quality of a wireless channel; Determine, based on the current reinforcement learning model, a second target action corresponding to the second state, wherein the second target action includes a parameter value of a parameter affecting a physical layer rate; Based at least on the second target action, sending second data to a third device; After sending the second data, measuring a corresponding first throughput rate; Using the first throughput rate as a first reward, updating parameter values ​​of model parameters of the current reinforcement learning model; Repeat the above steps until the iteration cutoff condition is met to obtain the trained reinforcement learning model.

3. The method according to claim 1, characterized in that The method further comprises: After sending the first data, measuring a corresponding second throughput rate; The second throughput rate is used as a second reward to update parameter values ​​of model parameters of the trained reinforcement learning model.

4. The method according to claim 3, characterized in that The reward is modified through the value network to obtain a modified reward; the reward is the first reward or the second reward; The modified reward is input into a loss function, and the model parameters of the reinforcement learning model are updated according to the gradient of the loss function.

5. The method according to claim 1, characterized in that The method further comprises: Determining at least one third target action based on the first target action and a preconfigured physical layer rate list; The at least one third target action is used to resend the first data to the first device when the first data fails to be sent.

6. The method according to claim 1, characterized in that The method further comprises: Based on the first target action, the first data is coded and modulated, thereby sending the first data to the first device; wherein the first target action at least includes: an MCS index, a bandwidth, a protection interval, and a spatial stream number.

7. The method according to any one of claims 1 to 6, characterized in that: The method is applied to a second device, where the second device and the first device are Wi-Fi devices.

8. A data processing device, characterized in that: include: A first acquisition module is configured to acquire a first state, wherein the first state includes measurement data for reflecting the quality of a wireless channel; wherein the measurement data includes state data at a current moment and state data at a previous moment; and the measurement data includes at least one of the following: received signal strength indication, error vector magnitude, signal-to-noise ratio, retransmission rate, and packet loss rate; A first determination module is configured to determine a first target action corresponding to the first state based on a trained reinforcement learning model, wherein the first target action includes a parameter value of a parameter affecting a physical layer rate; wherein the first target action is used to send first data to a first device, and the trained reinforcement learning model is obtained by training based on a second state and a first reward for a second target action corresponding to the second state, and the first reward is obtained based on at least a first throughput rate achieved when the second target action sends second data; The first sending module is configured to encode and modulate the first data based on the first target action, so as to send the first data to the first device.

9. The device according to claim 8, characterized in that The device also includes: A second acquisition module is configured to acquire a second state, wherein the second state includes measurement data reflecting the quality of the wireless channel; A second determination module is configured to determine a second target action corresponding to the second state based on the current reinforcement learning model, wherein the second target action includes a parameter value of a parameter affecting a physical layer rate; A second sending module, configured to send second data to a third device based at least on the second target action; a first measuring module, configured to measure a corresponding first throughput rate after sending the second data; an updating module, configured to update a parameter value of a model parameter of the reinforcement learning model using the first throughput rate as a first reward; The iteration module is configured to trigger the above modules until the iteration cutoff condition is met to obtain the trained reinforcement learning model.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.