Flow control method, system and device based on channel data and storage medium

Abstract

The invention discloses a flow control method, system and device based on channel data and a storage medium, and relates to the field of network flow control. The method comprises the following steps: inputting air interface layer statistical data and protocol stack layer statistical data based on a reinforcement learning algorithm, training in an offline training mode, and applying the training data to flow control reasoning under an online condition to control the size of a sending window; predicting the size of an optimal sending window through reinforcement learning; and a channel estimation and flow algorithm is adopted, and based on a reinforcement learning result, the to-be-sent data volume of the sending end is controlled, so that flow control is realized. Real-time data and a dynamic mapping rule are adopted, the size of a sending window and sending time delay are dynamically adjusted, packets are automatically merged or segmented, and dynamic flow control is achieved.

Description

Technical Field

[0001] The present invention relates to the field of network flow control, and in particular to a flow control method, system, device and storage medium based on channel data. Background Art

[0002] With the continuous updating of technologies such as mobile Internet, Internet of Things, big data, systems, equipment, storage media, and cloud computing, the number of network users and network scale are growing rapidly. In the face of the explosive growth of network traffic, wireless communication technology is also widely used.

[0003] Wireless communication technology has also seen great development, such as WiFi (Wireless Fidelity), Bluetooth, 5G (5th-Generation Mobile Communication Technology), Super Low Bandwidth (SLB), and Super Low Energy (SLE).

[0004] Due to the mutual crosstalk of wireless technologies in the same frequency band, interference from signal interference sources (such as microwave ovens, wireless phones and other electronic devices), the multipath effect of wireless signals caused by shielding objects such as buildings, and the weakening of wireless signals by external harsh environments, air interface data transmission will fail or error, resulting in fluctuations in transmission rate and transmission delay. Such fluctuations in transmission delay and transmission rate will lead to uncontrollable situations in application development. Therefore, it is necessary to study related methods of flow control.

[0005] In network transmission, when the network is congested, traditional flow control solutions mostly use flow control algorithms based on fixed mapping rules, such as CUBIC (an algorithm that uses a cubic function as a congestion window), GCC (Generic Congestion Control), BBR (Bottleneck Bandwidth and Round-trip propagation time, a congestion control algorithm based on bandwidth and delay feedback), etc., which cannot meet the low latency requirements under high throughput conditions. In recent years, researchers have used machine learning technology to optimize the flow control problem in high-throughput data transmission.

[0006] The current common flow control algorithm is only based on protocol stack layer data, such as transmission rate, packet loss rate, etc. It cannot be associated with air interface information, such as RSSI (Received Signal Strength Indication) and MCS (Mymova Checkin System).

[0007] The existing flow control algorithm is based on fixed mapping rules. When the network transmission status changes, it will not perform real-time statistics on the current situation; it is sent based on the existing data packet length, and the protocol stack layer does not perform additional processing on the data packet. Therefore, it is difficult to obtain the best flow control effect. Summary of the invention

[0008] In view of the defects existing in the prior art, the purpose of the present invention is to provide a flow control method, system, device and storage medium based on channel data, which can adopt real-time data and dynamic mapping rules to dynamically adjust the sending window size, sending delay and automatically merge or split packets to achieve dynamic flow control.

[0009] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0010] In a first aspect, the present application provides a flow control method based on channel data, comprising the following steps:

[0011] Based on the reinforcement learning algorithm, the statistical data of the air interface layer and the protocol stack layer are input, and training is performed through offline training. The training data is used for flow control reasoning in the online case to control the sending window size; the optimal sending window size is predicted through reinforcement learning;

[0012] Channel estimation and flow algorithms are used to control the amount of data to be sent by the sender based on the results of reinforcement learning, thereby achieving flow control.

[0013] Based on the flow control method described in the first aspect, the method further includes the following steps:

[0014] Automatically adjust the data size of the transmitted data based on the air interface layer statistical data.

[0015] Based on the flow control method described in the first aspect, the method further includes the following steps:

[0016] When the network condition is good, the packet data to be sent is merged and then sent;

[0017] When the network condition is poor, the data to be sent will be split into smaller packets and then sent.

[0018] Based on the flow control method described in the first aspect, the method further includes the following steps:

[0019] The transmission rate and packet loss rate in the protocol stack layer data are extracted, and reinforcement learning training is performed to obtain the relationship between the protocol stack layer data and the sending window and sending delay.

[0020] Based on the flow control method described in the first aspect, the method further includes the following steps:

[0021] The received signal strength indication RSSI and modulation and coding strategy MCS in the lower-layer air interface information are extracted, and reinforcement learning training is performed to obtain the relationship between the lower-layer air interface related information and the sending window and sending delay.

[0022] In a second aspect, the present application provides a flow control system based on channel data, comprising:

[0023] The reinforcement learning module is used to: input air interface layer statistics and protocol stack layer statistics based on the reinforcement learning algorithm, perform training through offline training, and use the training data for flow control reasoning in online situations to control the sending window size; predict the optimal sending window size through reinforcement learning;

[0024] The module for controlling the amount of data to be sent is used to: use channel estimation and flow algorithm, based on the results of reinforcement learning, control the amount of data to be sent by the sender to achieve flow control.

[0025] On the basis of the flow control system described in the second aspect, the system further includes:

[0026] The data size adjustment module is used to automatically adjust the data size of the transmission data according to the air interface layer statistical data.

[0027] On the basis of the flow control system described in the second aspect, the system further includes:

[0028] The merging module is used to merge the packet data to be sent and then send it when the network condition is good;

[0029] The splitting module is used to split the data to be sent into smaller packets and then send them when the network condition is poor.

[0030] In a third aspect, the present application provides a flow control device, comprising a processor, a memory, and a flow control program stored in the memory and executable by the processor, wherein when the flow control program is executed by the processor, the steps of the flow control method described in the first aspect are implemented.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium, on which a flow control program is stored, wherein when the flow control program is executed by a processor, the steps of the flow control method described in the first aspect are implemented.

[0032] Compared with the prior art, the advantages of the present invention are:

[0033] The present invention adopts real-time data and dynamic mapping rules to dynamically adjust the sending window size, the sending delay and automatically merge or split the packets, thereby performing dynamic flow control. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 is a flow chart of a flow control method based on channel data in an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a state machine transition of a Markov decision process in an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of the architecture of reinforcement learning training in an embodiment of the present invention;

[0038] Figure 4 This is a diagram of the main network topology related to training in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0041] First, see Figure 1 As shown, an embodiment of the present invention provides a flow control method based on channel data, comprising the following steps:

[0042] Based on the reinforcement learning algorithm, the statistical data of the air interface layer and the protocol stack layer are input, and training is performed through offline training. The training data is used for flow control reasoning in the online case to control the sending window size; the optimal sending window size is predicted through reinforcement learning;

[0043] The channel estimation and flow algorithm is based on the results of reinforcement learning to control the amount of data to be sent by the sender, thereby achieving the purpose of controlling the flow.

[0044] This layer in the embodiment of the present invention is the protocol stack layer, for example: Bluetooth protocol stack / Star Flash protocol stack / IP protocol stack, etc.; the air interface layer refers to the physical layer, for example: Bluetooth air interface, WiFi, Star Flash air interface, 2.4G air interface, etc.

[0045] The embodiment of the present invention includes the air interface related indicators of the wireless device into the flow control related parameters, automatically adjusts the data size of the transmission data according to the air interface statistical data, and processes it according to the network conditions. For example, when the network conditions are good, the packet data to be sent are merged and then sent; when the network conditions are poor, the data to be sent is split into smaller packets and then sent.

[0046] The embodiment of the present invention performs flow control based on pre-training of reinforcement learning, which is mainly used for flow control under network changes. Through pre-training data, a priori model is formed, and in actual situations, the priori model is used to control congestion of the changing network.

[0047] 1. Extract protocol stack layer data, such as transmission rate, packet loss rate and other protocol stack layer data, conduct reinforcement learning training, and obtain the relationship between protocol stack layer data and sending window and sending delay;

[0048] 2. Extract the lower-layer air interface information, such as RSSI (Received Signal Strength Indication), MCS (Mymova Checkin System), and other information, perform reinforcement learning training, and extract the relationship between the lower-layer air interface related information and the sending window and sending delay;

[0049] 3. Save the training results as offline data. During operation, periodically extract the statistical information of the protocol stack layer and the lower layer. Dynamically adjust the sending window and sending delay through offline data and automatically merge or split the packets to achieve the purpose of flow control.

[0050] In summary, the existing flow control algorithm is based on fixed mapping rules; while the embodiment of the present invention adopts real-time data and dynamic mapping rules to dynamically adjust the sending window size, sending delay and automatically merge or split packets, thereby performing dynamic flow control.

[0051] The embodiment of the present invention is based on a reinforcement learning algorithm, inputs air interface layer statistical data and protocol stack layer statistical data, performs training through an offline training method, and uses the training data for flow control reasoning in an online situation, mainly controlling the sending window size.

[0052] The channel estimation and flow algorithm are used to control the amount of data to be sent by the sender (D) based on the results of reinforcement learning, so as to achieve the purpose of controlling the flow.

[0053] First, because there is no strong correlation between the velocity fluctuation rate of the physical layer and the velocity at the previous moment, it can be considered that the previous and subsequent states are independent and can be modeled through the Markov decision process, which is a sequential decision problem.

[0054] See also Figure 2 As shown in Figure 2, the Markov decision process consists of a four-tuple consisting of state (S), action (A), reward function (R) and state transfer function (T). The state transition is shown in Figure 2. Figure 1 As shown, in Figure 1 In the environment shown, in a specific state space, in the time sequence t={0,1,2,…,n} in reinforcement learning, at any time t, the next state St+1 is only related to the current state St, but not to the past state. St in any time state will interact with the environment and produce an action At belonging to the action space A.

[0055] Then, the reinforcement learning agent will move to the next state St+1 and generate the reward function value R of this stage. The ultimate goal of learning is to learn the best actions based on past experience in order to maximize the cumulative reward return.

[0056] The state space is defined as: Under the current conditions, the data values ​​that can be counted are: packet loss rate (l), transmission delay (d), transmission delay variance (j), actual transmission rate (v), and instantaneous Rssi value (i).

[0057] All indicators are first normalized and then used as the input of the model to avoid interference from outliers that may affect the normal training of the neural network and gradient updates.

[0058] Now, the specific implementation of the normalization method is described by taking the packet loss rate as an example.

[0059] Assume that the sequence is: l = {l1,l2,l3,l max ,…,l n},

[0060] Then the normalization method is: l_n = {l1 / l max ,l2 / l max ,l3 / lmax ,…,l n / l max}.

[0061] The same goes for other indicators.

[0062] Therefore, the state input of all indicators after data normalization is:

[0063] S n = {l n ,d n ,j n ,v n ,i n}.

[0064] The reward function design refers to the reward function mechanism of the Pensieve algorithm and the OnRL algorithm, and designs a reward function based on real-time video optimization, mainly based on the following indicators: actual transmission rate (V), transmission delay (d), and continuous variance of actual transmission rate (vi-vi-1).

[0065] As for the actual results:

[0066] The transmission rate is a positive factor (reward factor) because for data transmission, the faster the transmission rate, the better;

[0067] Transmission delay (penalty factor), because for data transmission, the longer the delay, the worse the transmission quality, so the smaller the better;

[0068] For the actual transmission rate continuous variance (reward factor), the more stable the speed, the better; therefore, the smaller the absolute value, the better.

[0069] To sum up the above, the weighted reward function is:

[0070]

[0071] Wherein, N represents the length of a data packet in an input state, w1>0, w2>0, w3>0. In the physical layer system, this application sets a set of optimal combinations after combining and comparing multiple parameters. For practical use, the values ​​of w1, w2, and w3 are set to 30, 7, and 2, respectively.

[0072] Action Space The continuous action space is defined as the transmission prediction value of the transmission layer, and the output range is A = [a_ min ,a_ max ], and the specific value is any discrete value in this space.

[0073] The state space is input to the neural network model. The algorithm will generate an action value a of the optimal amount of data to be sent (D) by the sender according to the current strategy. This action value will be transmitted to the sender as the video sending rate in the next stage.

[0074] The model then generates a reward function value for this feedback to determine the benefit of the previous action;

[0075] If the return value is large, the model will tend to generate such a better strategy;

[0076] If the benefit value is small or negative, the model's parameter training will avoid generating similar actions.

[0077] The above stages are repeated over and over again. After training a certain size of data set and gaining more experience, it will evolve into a prediction model that can reasonably predict the amount of data to be sent by the sender (D).

[0078] The optimization goal of this model is to minimize the difference between the absolute value of the amount of data to be sent (D) predicted by the reinforcement learning algorithm and the actual bandwidth. The objective function T can be expressed as:

[0079] T=Min(|hA(argmax(L(p)))|)

[0080] Among them, b represents the network bandwidth, L(P) represents the probability matrix predicted by the last layer of the neural network of the reinforcement learning model, and A is the action space of the reinforcement learning model.

[0081] The PPO (Proximal Policy Optimization) algorithm is used as the update strategy of the model to ensure that the step size is not too large, which will lead to a large difference between the new and old strategies and cause the model to be difficult to converge.

[0082] Training architecture, such as Figure 3 As shown in the figure, through experience and testing, the important parameter configurations in the neural network are obtained: the neural network structure type adopts the fully connected layer, the input state fusion method adopts the first fusion method, the activation function adopts LeakyReLu, and the batch size adopts 64.

[0083] The main network topologies related to training, such as Figure 4 As shown, the training data is directly converted into inference parameters and written into the code. Through the network statistical environment, the size of the window to be sent is controlled, thereby automatically adjusting the amount of data sent per unit time.

[0084] In a second aspect, an embodiment of the present invention provides a flow control system based on channel data, including:

[0085] The reinforcement learning module is used to: input air interface layer statistics and protocol stack layer statistics based on the reinforcement learning algorithm, perform training through offline training, and use the training data for flow control reasoning in online situations to control the sending window size; predict the optimal sending window size through reinforcement learning;

[0086] The module for controlling the amount of data to be sent is used to: use channel estimation and flow rate algorithm, based on the result of reinforcement learning, control the amount of data to be sent by the sender, and realize flow control;

[0087] A data size adjustment module is used to automatically adjust the data size of the transmission data according to the statistical data of the air interface layer;

[0088] The merging module is used to merge the packet data to be sent and then send it when the network condition is good;

[0089] The splitting module is used to split the data to be sent into smaller packets and then send them when the network condition is poor.

[0090] Among them, the functional implementation of each module in the above-mentioned flow control system corresponds to the various steps in the above-mentioned flow control method embodiment, and its functions and implementation processes will not be repeated here one by one.

[0091] In a third aspect, an embodiment of the present invention provides a flow control device, which includes a processor, a memory, and a flow control program stored in the memory and executable by the processor, wherein when the flow control program is executed by the processor, the steps of the flow control method described in the first aspect are implemented.

[0092] In an embodiment of the present application, the flow control device may include a processor, a memory, a communication interface, and a communication bus.

[0093] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0094] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the AAAA device, and the interface used to realize the interconnection between the AAAA device and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display (Display), a keyboard (Keyboard), etc.

[0095] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0096] The processor may be a general-purpose processor, and the general-purpose processor may call the AAAA program stored in the memory and execute the AAAA method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the AAAA program is called may refer to the various embodiments of the flow control method of the present application, and will not be repeated here.

[0097] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0098] A flow control program is stored on the computer-readable storage medium of the present application, wherein when the flow control program is executed by a processor, the steps of the flow control method described in the first aspect are implemented.

[0099] Among them, the method implemented when the flow control program is executed can refer to the various embodiments of the flow control method of the present application, and will not be repeated here.

[0100] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0101] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0102] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0103] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0104] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0106] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A flow control method based on channel data, characterized in that: The following steps are involved: Based on the reinforcement learning algorithm, the statistical data of the air interface layer and the protocol stack layer are input, and training is performed through offline training. The training data is used for flow control reasoning in the online case to control the sending window size; the optimal sending window size is predicted through reinforcement learning; Channel estimation and flow algorithms are used to control the amount of data to be sent by the sender based on the results of reinforcement learning, thereby achieving flow control.

2. The flow control method based on channel data according to claim 1, characterized in that: The following steps are also included: Automatically adjust the data size of the transmitted data based on the air interface layer statistical data.

3. The method for flow control based on channel data according to claim 1, characterized in that: The following steps are also included: When the network condition is good, the packet data to be sent is merged and then sent; When the network condition is poor, the data to be sent will be split into smaller packets and then sent.

4. The method for flow control based on channel data according to claim 1, characterized in that: The following steps are also included: The transmission rate and packet loss rate in the protocol stack layer data are extracted, and reinforcement learning training is performed to obtain the relationship between the protocol stack layer data and the sending window and sending delay.

5. The method for flow control based on channel data according to claim 1, characterized in that: The following steps are also included: The received signal strength indication RSSI and modulation and coding strategy MCS in the lower-layer air interface information are extracted, and reinforcement learning training is performed to obtain the relationship between the lower-layer air interface related information and the sending window and sending delay.

6. A flow control system based on channel data, characterized in that: include: The reinforcement learning module is used to: input air interface layer statistics and protocol stack layer statistics based on the reinforcement learning algorithm, perform training through offline training, and use the training data for flow control reasoning in online situations to control the sending window size; predict the optimal sending window size through reinforcement learning; The module for controlling the amount of data to be sent is used to: use channel estimation and flow algorithm, based on the results of reinforcement learning, control the amount of data to be sent by the sender to achieve flow control.

7. The channel data-based flow control system according to claim 6, further comprising: The data size adjustment module is used to automatically adjust the data size of the transmission data according to the air interface layer statistical data.

8. The channel data based flow control system according to claim 6, further comprising: The merging module is used to merge the packet data to be sent and then send it when the network condition is good; The splitting module is used to split the data to be sent into smaller packets and then send them when the network condition is poor.

9. A flow control device, characterized in that: The flow control device includes a processor, a memory, and a flow control program stored in the memory and executable by the processor, wherein when the flow control program is executed by the processor, the steps of the flow control method as described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a flow control program, wherein when the flow control program is executed by a processor, the steps of the flow control method according to any one of claims 1 to 5 are implemented.

Images