A method for constructing a communication decision model based on Markov process

Abstract

The present invention belongs to the field of communication network transmission in poor channels, and discloses a method for constructing a communication decision model based on a Markov process, comprising: obtaining the real-time communication network environment of a communication terminal; constructing a data transmission model based on the Markov decision process, using the obtained real-time communication network environment as a state set, using the corresponding executable data transmission methods as an action set, and combining a real-time network environment detection module to construct a decision model for the data transmission method. Compared with traditional methods for improving the network environment, the present invention achieves new breakthroughs in the real-time performance of network detection and in achieving complete information or coverage of all important information content in poor channels. It realizes the correct selection of data transmission methods when the communication network quality is poor, thereby ensuring the stability of the transmission quality of the communication content during the communication process and avoiding channel waste.

Description

Technical Field

[0001] The present invention belongs to the field of communication network transmission in a bad channel, and in particular relates to a method for constructing a communication decision model based on a Markov process. Background Art

[0002] Ensuring reliable information transmission has always been a critical issue in the communications field. With the development of modern communications, information transmission faces even greater challenges. High-quality information transmission places high demands on information integrity and stability. Increasing network bandwidth and optimizing the communication network environment are urgent needs. However, given the complex geographical and ecological environment, improving network environments is difficult and costly. Choosing the right data transmission method has become a new path to solving this information transmission problem.

[0003] At present, real-time measurement of network bandwidth is still challenging. The mainstream technology calculates the throughput by testing the network over a period of time. This is insufficient to ensure real-time information transmission. It also makes the corresponding communication strategy model unable to adapt to the current development of communications, which may lead to distortion or packet loss during information transmission.

[0004] In today's highly intelligent world, the choice of information transmission methods presents even greater challenges. First, the autonomous interaction between client and server requires more than simply relying on server instructions for operation and feedback. The client must play a more crucial role in the communication process, autonomously sensing the working environment and providing feedback to the server to ensure stable and timely changes in information transmission methods during communication. This poses significant challenges to the construction of communication strategy models. Summary of the Invention

[0005] The present invention provides a method for constructing a communication decision model based on a Markov process, which is mainly used to ensure that the information to be transmitted can be reached in complex and harsh environments. The information itself can undergo modal conversion depending on the quality of the channel, so as to solve the problem that the existing communication network quality detection and information transmission mode selection are insufficient in real time and simplicity.

[0006] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0007] The present invention is a method for constructing a communication decision model based on a Markov process, which specifically includes the following steps:

[0008] Step 1: Acquire the real-time communication network environment quality, use the acquired real-time communication network quality as a state set, and modify the communication network quality obtained by the test from a visual state to a readable state;

[0009] Step 2: Define the state set as S and introduce the reward function R. Finally, use the Markov decision model to determine whether the cumulative reward expectation measurement strategy π is the optimal solution.

[0010] Step 3: Define the action set A in the decision process of constructing the Markov decision model. The optimal strategy is to represent the correct mapping relationship between the state set S and the action set A.

[0011] Step 4. Define the value function V(π) for the cumulative reward expectation measurement strategy π and calculate the cumulative reward expectation when the starting state set S is given and the corresponding action set A is used according to the cumulative reward expectation measurement strategy π;

[0012] Step 5. Compare V(π) values ​​to obtain the optimal strategy π * , that is, the data transmission type selection model and executes the corresponding data transmission method according to the optimal strategy.

[0013] A further improvement of the present invention is that: in step 1, obtaining the quality of the real-time communication network environment and performing a readable operation are specifically: installing a communication network testing tool on the client and the server, and performing operations on the client to obtain the real-time communication network quality, and dividing the real-time communication network quality into intervals according to the communication network environment.

[0014] A further improvement of the present invention is that: the network testing tool is an iperf3 tool, the iperf3 tool is installed on the server and the client, the server is set to the answering mode, and the client executes the sending and testing network performance instructions. In order to ensure that the obtained communication network quality data is real-time, a high frequency is set for testing, and then the real-time communication network quality data is pulled through the server. Since the real-time communication network quality data obtained by the network testing tool iperf3 cannot be read, the real-time communication network quality results read by the iperf3 tool are read by writing an executable file.

[0015] A further improvement of the present invention is that the acquired real-time communication network quality is divided into a low-quality interval, a medium-quality interval, and a high-quality interval based on the standards of bandwidth less than 500Kb / s, greater than 500Kb / s and less than 2Mb / s, and greater than 2Mb / s. When the communication bandwidth is less than 500Kb / s, text transmission is performed; when the communication bandwidth is greater than 500Kb / s and less than 2Mb / s, picture transmission is performed; and when the communication bandwidth is greater than 2Mb / s, video transmission is performed.

[0016] A further improvement of the present invention is that in step 3, three data transmission modes, namely video transmission, picture transmission, and text transmission, are defined as action set A, and strategy π is defined as a one-to-one mapping of different quality communication network environments to different data transmission modes.

[0017] Establishing a prototype model using a Markov process: A reinforcement learning process that satisfies the Markov property is called a Markov decision process. The quality of the communication network and the selected data transmission method in the communication decision model are defined as the state set S and action set A in the Markov decision process.

[0018] The quality of the communication network is defined as the state set S in a Markov decision process. During reinforcement learning, the probability transitions of the state set S itself are ignored. As the state changes over time, that is, as the communication network environment changes, actions in the action set A are randomly executed, using different data types to transmit data. This mapping between state and action is defined as policy π. Since time and action are both sets, policy π is also not fixed. The probability policy for taking action a given the input state s is as follows:

[0019] π(a|s)=p[A t =a|S t =s].

[0020] In the reinforcement learning process, under different communication network quality environments, the server can perform the correct selection of information transmission type, which requires the definition of the state value function V π (s) and the action-value function Q π (s,a).

[0021] The state value function is the expected return that can be obtained by following the strategy π when the state value function is state s. The mathematical expression is as follows:

[0022] V π (s)=E π [G t |S t =s]

[0023] Action-value function Q π (s,a) represents the expected return obtained by following the strategy π in the Markov decision process. The mathematical expression is as follows:

[0024] Q π (s,a)=E π [G t ∣S t =s,A t =a]

[0025] After defining different communication network qualities corresponding to different data transmission modes, the optimal strategy is calculated using the Bellman equation form of two functions. The optimal strategy refers to the situation in which V π (S)≥V π′ (S), denoted as π* (s). The specific calculation method is as follows: First calculate the Bellman equation form of the two value functions:

[0026] Q π (s,a)=E(R a (s,s′)+γV π (s′)|s t =s,a t =a)

[0027] V π (s)=E(R a (s,s′)+γV π (s′)|s t =s)

[0028] By solving the two Bellman equations above, we can calculate the optimal strategy and obtain the optimal state value function and the optimal dynamic function. Marginalize the optimal state value function and maximize Q * (s,a), we can find the optimal strategy:

[0029]

[0030] Through the Markov decision process, the communication between the client and the server always follows the optimal strategy, ensuring that during real-time communication, the terminal can choose the data transmission method based on the quality of the real-time communication network, ensuring the stability of data transmission.

[0031] The beneficial effects of the present invention are as follows: the present invention is a communication strategy model based on the Markov decision process, so in the process of formulating the communication strategy, it is not affected by the past state, and only the correct rewards and feedback are given through the current communication network environment factors, which ensures that the execution of the communication strategy is highly real-time, which coincides with the reliable transmission of information in the field of modern communication transmission. The method of adjusting the data transmission mode and improving the network bandwidth by the server autonomously following the communication strategy greatly reduces the construction and operation costs. In addition, the improvement and use of the iperf3 tool in the present invention also provides a new idea for how to test the real-time network bandwidth. Compared with traditional methods such as measuring communication bandwidth and optimizing communication quality, a new solution is provided in terms of detecting network communication bandwidth and solving data transmission problems under poor communication quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of constructing a communication model according to an embodiment of the present invention.

[0033] Figure 2 It is a real-time communication network quality graph obtained by improving the iperf3 tool in an embodiment of the present invention.

[0034] Figure 3 is a reward cycle diagram in the Markov decision process in an embodiment of the present invention.

[0035] Figure 4 It is a decision model diagram constructed by the method of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

[0038] like Figure 1 As shown, the present invention is a method for constructing a communication decision model based on a Markov process, and the method for constructing the communication decision model specifically includes the following steps:

[0039] Step 1: Obtain the real-time communication network environment quality, use the obtained real-time communication network quality as a state set, and modify the communication network quality obtained from the test from a visual state to a readable state, that is, install the communication network test tool on the client and server, and operate on the client to obtain the real-time communication network quality, and divide the real-time communication network quality into intervals according to the communication network environment.

[0040] like Figure 2 As shown, the network testing tool of the present invention is an iperf3 tool. The iperf3 tool is installed on the server and the client, the server is set to the answering mode, and the sending and testing network performance instructions are executed on the client. In order to ensure that the obtained communication network quality data is real-time, a high frequency is set for testing, and then the real-time communication network quality data is pulled through the server. Since the real-time communication network quality data obtained by the network testing tool iperf3 tool cannot be read, an executable file is written, which is a bash script. The awk command is used to extract the data in the 6th column read by iperf3, and the result is assigned to the variable speed and output.

[0041] Read the real-time communication network quality results read by the iperf3 tool.

[0042] The obtained real-time communication network quality is divided into low quality interval, medium quality interval and high quality interval according to the standards of bandwidth less than 500Kb / s, greater than 500Kb / s and less than 2Mb / s, and greater than 2Mb / s. When the communication bandwidth is less than 500Kb / s, text transmission is performed; when the communication bandwidth is greater than 500Kb / s and less than 2Mb / s, picture transmission is performed; when the communication bandwidth is greater than 2Mb / s, video transmission is performed.

[0043] Step 2: Define the intervals divided in step 1 as different states, set as high quality, heavy quality, and low quality, form a state set defined as S, and introduce the reward function R. Finally, use the Markov decision model to determine whether the cumulative reward expectation measurement strategy π is the optimal solution;

[0044] In step 2, the calculation formula of the reward function Rs is:

[0045] Rs=E[Rt+1|St=S];

[0046] Among them, S represents the state set, including high-quality state, medium-quality state and low-quality state, Rt and St represent the state and feedback at time t respectively.

[0047] Step 3: Define the action set A in the decision-making process of constructing the Markov decision model. The optimal strategy is to represent the correct mapping relationship between the state set S and the action set A. The three data transmission modes, namely video transmission, image transmission, and text transmission, are defined as the action set A. The strategy π is defined as a one-to-one mapping of different communication network environments with different data transmission modes.

[0048] Step 4. Define the value function V(π) for the cumulative reward expectation measurement strategy π and calculate the cumulative reward expectation when the starting state set S is given and the corresponding action set A is used according to the cumulative reward expectation measurement strategy π;

[0049] Step 5. Compare V(π) values ​​to obtain the optimal strategy π * , that is, the data transmission type selection model and executes the corresponding data transmission method according to the optimal strategy.

[0050] like Figure 3 As shown, a prototype model is established using a Markov decision process: a reinforcement learning process that satisfies the Markov property is called a Markov decision process. The communication network quality (low, medium, and high) and the selected data transmission methods (video, image, and command transmission) in the communication decision model are defined as the state set S and action set A in the Markov decision process.

[0051] The quality of the communication network is defined as the state set S in a Markov decision process. During reinforcement learning, the probability transitions of the state set S itself are ignored. As the state changes over time, that is, as the communication network environment changes, actions in the action set A are randomly executed, using different data types to transmit data. This mapping between state and action is defined as the policy π. Since time and action are both sets, the policy π is also not fixed. The probability policy for taking action a given the input state s is as follows:

[0052] π(a|s)=p[A t =a|S t =s].

[0053] In the reinforcement learning process, under different communication network quality environments, the server can perform the correct selection of information transmission type, which requires the definition of the state value function V π (s) and the action-value function Q π (s,a).

[0054] The state value function is the expected return that can be obtained by following the strategy π when the state value function is state s. The mathematical expression is as follows:

[0055] V π (s)=E π [G t |S t =s]

[0056] Action-value function Q π (s,a) represents the expected return obtained by following the strategy π in the Markov decision process. The mathematical expression is as follows:

[0057] Q π (s,a)=E π [G t ∣S t =s,A t =a]

[0058] After defining different communication network qualities corresponding to different data transmission modes, the optimal strategy is calculated using the Bellman equation form of two functions. The optimal strategy refers to the situation in which V π (S)≥V π′ (S), denoted as π * (s). The specific calculation method is as follows: First calculate the Bellman equation form of the two value functions:

[0059] Q π (s,a)=E(R a (s,s′)+γV π (s′)|s t=s,a t =a)

[0060] V π (s)=E(R a (s,s′)+γV π (s′)|s t =s)

[0061] By solving the two Bellman equations above, we can calculate the optimal strategy. In this example, the optimal strategy is to transmit text when the communication bandwidth is less than 500Kb / s, transmit pictures when the communication bandwidth is greater than 500Kb / s and less than 2Mb / s, and transmit videos when the communication bandwidth is greater than 2Mb / s. We can also obtain the optimal state value function and the optimal dynamic function. The optimal state value function is marginalized and the optimal dynamic function is obtained by maximizing Q * (s,a), we can find the optimal strategy judgment conditions:

[0062]

[0063] like Figure 4 Shown is a diagram of a model constructed by the method of the present invention.

[0064] The present invention is based on a communication strategy model based on a Markov decision process. Therefore, the process of formulating communication strategies is not affected by past states, and correct rewards and feedback are only provided based on current communication network environmental factors. This ensures that the execution of communication strategies is highly real-time, which coincides with the modern field of communication transmission regarding reliable information transmission. By having the server autonomously follow the communication strategy and adjust the data transmission method to improve network bandwidth, the method greatly reduces construction and operating costs. In addition, the improvement and use of the iperf3 tool in the present invention also provides a new entry point for how to test real-time network bandwidth.

Claims

1. A method for constructing a communication decision model based on a Markov process, characterized by: The method for constructing the communication decision model specifically includes the following steps: Step 1: Acquire the real-time communication network environment quality, use the acquired real-time communication network quality as a state set, and modify the communication network quality obtained by the test from a visual state to a readable state; Step 2: Define the state set as S and introduce the reward function R. Finally, use the Markov decision model to determine whether the cumulative reward expectation measurement strategy π is the optimal solution. Step 3: Define the action set A in the decision process of constructing the Markov decision model. The optimal strategy is to represent the correct mapping relationship between the state set S and the action set A. Step 4. Define the value function V(π) for the cumulative reward expectation measurement strategy π and calculate the cumulative reward expectation when the starting state set S is given and the corresponding action set A is used according to the cumulative reward expectation measurement strategy π; Step 5. Compare V(π) values ​​to obtain the optimal strategy π * , the optimal strategy π * Select a model for the data transmission type and execute the corresponding data transmission method according to the optimal strategy; In step 3, three data transmission modes, namely video transmission, picture transmission, and text transmission, are defined as an action set A, and a strategy π is defined as a one-to-one mapping between different communication network environments of different qualities and different data transmission modes.

2. The method for constructing a communication decision model based on a Markov process according to claim 1, characterized in that: In step 1, the real-time communication network environment quality is obtained and a readable operation is performed, specifically: a communication network test tool is installed on the client and the server, and an operation is performed on the client to obtain the real-time communication network quality, and the real-time communication network quality is divided into intervals according to the communication network environment.

3. The method for constructing a communication decision model based on a Markov process according to claim 2, characterized in that: The network testing tool is an iperf3 tool. The iperf3 tool is installed on the server and the client, the server is set to the listening mode, and the client executes the sending and testing network performance instructions. In order to ensure that the obtained communication network quality data is real-time, a high frequency is set for testing, and then the real-time communication network quality data is pulled through the server. Since the real-time communication network quality data obtained by the network testing tool iperf3 tool cannot be read, the real-time communication network quality results read by the iperf3 tool are read by writing an executable file.

4. The method for constructing a communication decision model based on a Markov process according to claim 2 or 3, characterized in that: The obtained real-time communication network quality is divided into low quality interval, medium quality interval and high quality interval according to the standards of bandwidth less than 500Kb / s, greater than 500Kb / s and less than 2Mb / s, and greater than 2Mb / s. When the communication bandwidth is less than 500Kb / s, text transmission is performed; when the communication bandwidth is greater than 500Kb / s and less than 2Mb / s, picture transmission is performed; when the communication bandwidth is greater than 2Mb / s, video transmission is performed.

5. The method for constructing a communication decision model based on a Markov process according to claim 1, characterized in that: In step 2, the calculation formula of the reward function Rs is: Rs=E[Rt+1|St=S]; Among them, S represents the state set, including high-quality state, medium-quality state and low-quality state, Rt and St represent the state and feedback at time t respectively.

6. The method for constructing a communication decision model based on a Markov process according to claim 5, characterized in that: In step 4, the optimal value function in a certain state is determined. The optimal value function is the size of the different value functions brought about by all different action sets A executed in the corresponding state S. The calculation formula is as follows: V * (s)=maxV π (s) The Bellman equation for the optimal value function is as follows: Where γ represents the discount factor, which is a real number between 0 and 1. P is the transition state matrix function, which is the probability function of the state S changing. The optimal strategy is obtained from the optimal value function. The optimal strategy calculation formula is as follows: For the state set S and the cumulative reward expectation measurement strategy π, we get: V * (s)=V π* (s)≥V π (s) Thus, we can get the data transmission mode of the action set A corresponding to the optimal strategy of the communication network quality of different state sets S, V π (s) is the state value function.

Images