Message retransmission method and device based on reinforcement learning and computer equipment

Abstract

The invention relates to the technical field of communication, and discloses a message retransmission method and device based on reinforcement learning and computer equipment, and the method comprises the steps: collecting the network state data of a node when a message sending task is received; the network state data comprises a signal-to-noise ratio, a bandwidth utilization rate, a packet loss rate and link quality of the current network; forming a state vector based on the network state data; obtaining target retransmission time through a reinforcement learning agent based on the state vector; and sending a message to the outer node based on the target retransmission time. Based on the reinforcement learning method, the retransmission time interval can be dynamically adjusted according to the network condition, and the optimal decision is made, so that invalid retransmission is reduced, and the transmission efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a message retransmission method, device and computer equipment based on reinforcement learning. Background Art

[0002] Automatic Repeat Request (ARQ) is a widely used error control mechanism that mainly indicates whether a data packet is successfully received through acknowledgment (ACK) and negative acknowledgment (NACK). If the receiver fails to receive the data packet correctly, it will send a NACK message to the sender to instruct it to resend the data packet. Although ARQ performs well under static network conditions, one of its major drawbacks is its fixed retransmission strategy. In a dynamically changing network environment, such as a mobile communication network with large signal strength fluctuations, ARQ may cause a large number of retransmission attempts, thereby increasing the risk of network congestion. In addition, ARQ is not friendly to real-time applications because it increases transmission delay, especially in the case of high packet loss rate.

[0003] In order to overcome the shortcomings of the fixed time interval retransmission mechanism, some algorithms have introduced a mechanism to dynamically adjust the retransmission time interval. For example, the Exponential Backoff Algorithm increases the retransmission interval exponentially according to the number of previous retransmission attempts after a conflict occurs. This method reduces the possibility of repeated transmissions in a short period of time, and thus alleviates network congestion to a certain extent. However, the exponential backoff algorithm still has some problems: first, it may cause large delays in the initial stage; second, when the network conditions change rapidly, the fixed exponential growth strategy may not be able to quickly adapt to the new network conditions, resulting in inappropriate retransmission intervals and affecting transmission efficiency. Summary of the invention

[0004] In view of this, the present invention provides a message retransmission method, apparatus and computer device based on reinforcement learning to solve the problem that the traditional message retransmission mechanism is difficult to give a suitable retransmission interval according to the actual situation of the network.

[0005] In a first aspect, the present invention provides a message retransmission method based on reinforcement learning, the method comprising:

[0006] When receiving a message sending task, collect the network status data of this node; the network status data includes the signal-to-noise ratio, bandwidth utilization, packet loss rate and link quality of the current network;

[0007] forming a state vector based on the network state data;

[0008] Based on the state vector, the target retransmission time is obtained through the reinforcement learning agent;

[0009] Send messages to outbound nodes based on target retransmission time.

[0010] The message retransmission method based on reinforcement learning provided by the embodiment of the present invention uses a reinforcement learning agent to form a state vector according to current network status data, such as signal-to-noise ratio, bandwidth utilization, packet loss rate, and link quality, and intelligently selects the target retransmission time. The method can dynamically adjust the retransmission time and strategy according to the real-time changes in network status data, which helps to avoid frequent retransmission of messages when the network is congested or unstable, thereby saving network resources. This intelligent decision-making method is more accurate and efficient than the traditional manual setting of retransmission strategies.

[0011] In an optional implementation, obtaining the target retransmission time through a reinforcement learning agent based on the state vector further includes:

[0012] Preprocessing the state vector to obtain the target state vector; the preprocessing includes normalization and regularization operations;

[0013] Through the reinforcement learning agent, the target state vector is processed to obtain the target retransmission time.

[0014] The message retransmission method based on reinforcement learning provided in the embodiment of the present invention can ensure that the data input to the reinforcement learning agent is in an appropriate range after normalization and regularization preprocessing operations, ensure that all input features are on the same scale, and avoid certain features dominating the decision-making process due to a large numerical range.

[0015] In an optional embodiment, the method further includes:

[0016] Set the maximum number of training steps;

[0017] When the current step number is not greater than the maximum training step number, the reinforcement learning agent is trained under the preset network fluctuation environment.

[0018] The message retransmission method based on reinforcement learning provided in the embodiment of the present invention can learn the optimal retransmission strategy under different network conditions by training in a preset network fluctuation environment, so that the agent can better adapt to the dynamic changes in the actual network and improve its stability and reliability under various network conditions. At the same time, setting the maximum number of training steps helps to control the length of the training process and prevent the agent from being over-trained or under-trained.

[0019] In an optional implementation, training a reinforcement learning agent in a preset network fluctuation environment includes:

[0020] Generate random numbers within the specified probability range, and generate random exploration probabilities based on the current number of steps and the maximum number of training steps;

[0021] Generate sample retransmission time based on the random number and the random exploration probability;

[0022] Under the preset network fluctuation environment, data is retransmitted according to the sample retransmission time to obtain the number of sample retransmissions and the sample time difference;

[0023] The network parameters of the reinforcement learning agent are updated according to the number of sample retransmissions and the sample time difference.

[0024] The message retransmission method based on reinforcement learning provided in an embodiment of the present invention generates a random exploration probability according to the current step number and the maximum training step number through random exploration, and generates a random number within a specified probability range, so that the intelligent agent can try a variety of different retransmission times, and update its network parameters based on the sample retransmission number and the sample time difference, which helps the intelligent agent to discover more efficient retransmission strategies, reduce unnecessary retransmission times, and improve the efficiency of message transmission.

[0025] In an optional implementation, generating a sample retransmission time according to a random number and a magnitude relationship of a random exploration probability includes:

[0026] When the random number is less than the random exploration probability, the reinforcement learning agent randomly selects the sample retransmission time within the preset range;

[0027] When the random number is greater than or equal to the random exploration probability, the sample retransmission time of the state vector corresponding to the preset network fluctuation environment of the reinforcement learning agent is obtained.

[0028] The message retransmission method based on reinforcement learning provided by the embodiment of the present invention can ensure that the agent has sufficient exploration opportunities during the training process and avoid falling into the local optimal solution by setting the random exploration probability and randomly selecting the sample retransmission time when the random number is less than the probability. At the same time, when the random number is greater than or equal to the random exploration probability, the agent uses the learned knowledge to select the sample retransmission time, thereby improving the training efficiency.

[0029] In an optional implementation, data is retransmitted according to the sample retransmission time under a preset network fluctuation environment to obtain the number of sample retransmissions and the sample time difference, including:

[0030] Under a preset network fluctuation environment, setting a sample retransmission time of the first node, and controlling the first node to send a message to the second node;

[0031] When the first node receives the reply message from the second node, the first node obtains the sample retransmission times and the sample time difference between the time of sending the message and the time of replying the message.

[0032] The message retransmission method based on reinforcement learning provided by the embodiment of the present invention can evaluate whether the retransmission time selected by the agent is effective by recording the number of sample retransmissions and the sample time difference, and update its strategy based on these feedbacks. This training method can ensure that the agent learns the optimal or suboptimal retransmission strategy under different network conditions.

[0033] In a second aspect, the present invention provides a message retransmission device for reinforcement learning, the device comprising:

[0034] The collection module is used to collect the network status data of the node when receiving the message sending task; the network status data includes the signal-to-noise ratio, bandwidth utilization, packet loss rate and link quality of the current network;

[0035] A forming module, for forming a state vector based on the network state data;

[0036] A calculation module, used to obtain a target retransmission time through a reinforcement learning agent based on a state vector;

[0037] The sending module is used to send messages to external nodes based on the target retransmission time.

[0038] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the message retransmission method based on reinforcement learning of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the message retransmission method based on reinforcement learning according to the first aspect or any corresponding embodiment thereof.

[0040] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the message retransmission method based on reinforcement learning according to the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1is a flow chart of a message retransmission method based on reinforcement learning according to an embodiment of the present invention;

[0043] Figure 2 is a flow chart of another message retransmission method based on reinforcement learning according to an embodiment of the present invention;

[0044] Figure 3 is a structural block diagram of a message retransmission device based on reinforcement learning according to an embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention (for a computer device having an input device and an output device). DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0047] In order to overcome the shortcomings of the fixed time interval retransmission mechanism in the traditional automatic repeat request (ARQ), some algorithms have introduced a mechanism to dynamically adjust the retransmission time interval. For example, the Exponential Backoff Algorithm increases the retransmission interval exponentially according to the number of previous retransmission attempts after a conflict occurs. This method reduces the possibility of recurrence in a short period of time, thereby alleviating network congestion to a certain extent, but the algorithm still has some problems. For example, it may cause large delays in the initial stage, and when the network conditions change rapidly, the fixed exponential growth strategy may not be able to quickly adapt to the new network conditions, resulting in inappropriate retransmission intervals and affecting transmission efficiency.

[0048] Therefore, although traditional retransmission mechanisms based on fixed or dynamically adjusted time intervals can provide certain error control capabilities in some cases, they generally lack the ability to adapt to changes in network conditions.

[0049] The embodiment of the present invention provides a message retransmission method based on reinforcement learning, which dynamically adjusts the retransmission time interval according to the network status through the reinforcement learning method, thereby providing a more flexible solution. Specifically, by modeling the network environment as a dynamic system and defining a reward function to encourage the agent to learn the optimal data packet retransmission strategy, reinforcement learning can adjust the retransmission parameters in real time. This method can make the best decision according to the actual status of the network, reduce invalid retransmissions, and improve transmission efficiency.

[0050] According to an embodiment of the present invention, an embodiment of a message retransmission method based on reinforcement learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] In this embodiment, a message retransmission method based on reinforcement learning is provided. Figure 1 is a flowchart of a message retransmission method based on reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0052] Step S101, when receiving a message sending task, collecting network status data of this node.

[0053] Specifically, when a communication node receives a task to send a message from itself to an external node, it does not immediately perform the sending operation, but waits for the optimal retransmission time T. That is, the task identification and queuing mechanism at this stage is different from the instant sending or fixed strategy-based retransmission arrangement in the traditional retransmission mechanism. By delaying the sending, the reinforcement learning algorithm can use the current network status to optimize the transmission timing, reduce unnecessary retransmissions, and thus improve the transmission efficiency.

[0054] Furthermore, the network status data includes the signal-to-noise ratio, bandwidth utilization, packet loss rate, and link quality of the current network.

[0055] Step S102: forming a state vector based on the network state data.

[0056] Specifically, the network status data collected in step S101 is combined into a state vector S, each element of the state vector S corresponds to a network status indicator, and the order of these elements in the vector is fixed. Compared with the traditional retransmission mechanism that only relies on a single or a few parameters to determine the retransmission strategy, the state vector can comprehensively consider multi-dimensional information, thereby making more sophisticated and comprehensive decisions.

[0057] Step S103, obtaining the target retransmission time through the reinforcement learning agent based on the state vector.

[0058] Specifically, the state vector S obtained in step S102 is used as the input of the reinforcement learning agent. The reinforcement learning agent includes one or more neural network layers for processing the input state vector and outputting corresponding decisions.

[0059] Furthermore, the agent may use one or more value functions to evaluate the pros and cons of different retransmission times. Based on the calculation results of the value function, the agent selects an optimal action, namely the target retransmission time T, to ensure that the target retransmission time is the action with the highest expected benefit in the current state.

[0060] In some optional implementations, the above step S103 further includes:

[0061] Preprocessing the state vector to obtain the target state vector; the preprocessing includes normalization and regularization operations;

[0062] Through the reinforcement learning agent, the target state vector is processed to obtain the target retransmission time.

[0063] The data preprocessing unit performs preprocessing operations such as normalization and regularization on the state vector S collected by the state collection unit. Specifically, normalization is used to ensure that all network state data, such as signal-to-noise ratio, bandwidth utilization, packet loss rate, link quality, etc., are within the same numerical range to avoid certain features with a larger numerical range from dominating the decision-making process. Regularization is used to further improve the generalization ability of the model to reduce the correlation between features or eliminate redundant information.

[0064] Step S104: Send a message to an external node based on the target retransmission time.

[0065] Specifically, after receiving the message sending task, the communication node will not immediately perform the sending operation. Once the target retransmission time T is reached, the communication node will immediately perform the sending operation and transmit the message to the target external node through the wireless channel.

[0066] In summary, the message retransmission method based on reinforcement learning provided in this embodiment, when a communication node receives a task to send a message to an external node, it will not immediately perform the sending operation, but first enter the waiting stage. This waiting stage is also different from the traditional mechanical timed sending or fixed retransmission arrangement, but dynamically waits for an optimal retransmission time calculated by the intelligent agent based on a specific network state. The network state is obtained from various network data, and the network data includes key indicators such as signal-to-noise ratio, bandwidth utilization, packet loss rate and link quality.

[0067] In order to obtain these network status data, the state collection unit inside the communication node is responsible for collecting and combining them into a state vector. Subsequently, the data preprocessing unit preprocesses the state vector, including normalization and regularization operations, to ensure that all input features are on the same scale to avoid some features dominating the decision process due to their large numerical range.

[0068] The preprocessed state vector is input into the reinforcement learning agent, which uses the previously learned experience to predict the future network status and calculate the optimal retransmission time accordingly. Finally, the communication node will perform the sending operation according to the optimal retransmission time calculated by the agent. The retransmission mechanism of reinforcement learning is adopted, and the retransmission parameters can be adjusted in real time. This method can make the best decision according to the actual situation of the network, reduce invalid retransmissions, and improve transmission efficiency.

[0069] In some optional embodiments, the method further comprises:

[0070] Set the maximum number of training steps;

[0071] When the current step number is not greater than the maximum training step number, the reinforcement learning agent is trained under the preset network fluctuation environment.

[0072] It is understandable that before applying the trained intelligent agent, the intelligent agent needs to be trained first. First, the training environment needs to be initialized, that is, the maximum number of training steps of the reinforcement learning intelligent agent, episode, is set, and the starting number of steps, k, is set to 0.

[0073] When the current number of steps is greater than the maximum number of training steps, the training process ends. If the current number of steps is not greater than the maximum number of training steps, the reinforcement learning agent continues to be trained in the preset network fluctuation environment.

[0074] In some optional implementations, training a reinforcement learning agent in a preset network fluctuation environment includes:

[0075] Generate random numbers within the specified probability range, and generate random exploration probabilities based on the current number of steps and the maximum number of training steps;

[0076] Generate sample retransmission time based on the random number and the random exploration probability;

[0077] Under the preset network fluctuation environment, data is retransmitted according to the sample retransmission time to obtain the number of sample retransmissions and the sample time difference;

[0078] The network parameters of the reinforcement learning agent are updated according to the number of sample retransmissions and the sample time difference.

[0079] like Figure 2 As shown, using a random number generator, a random number p between 0 and 1 can be generated based on the current step number or a fixed seed as a reference for the exploration probability of the simulated agent. The exploration probability of the agent at the current step number k is obtained from the function f(k). The function f(x) should be a monotonically non-increasing function with a value range of 0 to 1. For example, the following function can be used:

[0080] f(x)=1-e k-episode

[0081] Among them, episode is the maximum number of training steps, k is the current number of steps; according to the relationship between the random number and the random exploration probability, the sample retransmission time is generated, and the data is retransmitted according to the sample retransmission time; the environment will also give random fluctuations according to a certain strategy, simulate various scenarios in real network conditions, randomly interfere with the message transmission from the local node to the external node, and obtain the number of sample retransmissions and the sample time difference respectively. Finally, the environment will give the agent a reward value of r according to the strategy T given by the agent and the actual number of retransmissions cnt and the time difference Δt. The formula is as follows:

[0082] r k =g(T k ,cnt,Δt)

[0083] The agent uses the DDPG algorithm to update the neural network parameters and learn from the experience gained in this exploration. The steps are as follows:

[0084] (1) TD algorithm updates the value network:

[0085]

[0086] Among them, Q target is the target value, which is used to update the Q value of the corresponding message retransmission action in the deep learning algorithm. L(θ) is the value network loss function. π(s) is the policy network, the input is the state, and the output is the policy.

[0087] (2) The gradient ascent algorithm calculates the policy network gradient:

[0088]

[0089] in, is the policy network loss function.

[0090] (3) Update network parameters:

[0091]

[0092] in, θ is the policy network and value network parameters, and α is the learning rate. The value network scores the current action selected by the agent in the current state, and the policy network is the neural network ultimately used for agent decision-making. Its input is the current environment state, and its output is the best action.

[0093] In some optional implementations, generating a sample retransmission time according to a random number and a magnitude relationship between random exploration probabilities includes:

[0094] When the random number is less than the random exploration probability, the reinforcement learning agent randomly selects the sample retransmission time within the preset range;

[0095] When the random number is greater than or equal to the random exploration probability, the sample retransmission time of the state vector corresponding to the preset network fluctuation environment of the reinforcement learning agent is obtained.

[0096] like Figure 2 As shown in the figure, if the random number p is less than the current exploration probability, the agent randomly moves within the predefined time range [T min ,T max ] Randomly select a retransmission time T. If the random number p is greater than the current exploration probability, the agent enters the autonomous decision-making stage. The state collection unit and the data preprocessing unit collect the current network status, including signal-to-noise ratio, bandwidth utilization, packet loss rate, etc., to form a state vector S. The agent predicts the future state of the actual network based on the state vector S through the strategy network and gives the optimal sample retransmission time T under the current parameters.

[0097] In some optional implementations, data is retransmitted according to the sample retransmission time under a preset network fluctuation environment to obtain the number of sample retransmissions and the sample time difference, including:

[0098] Under a preset network fluctuation environment, setting a sample retransmission time of the first node, and controlling the first node to send a message to the second node;

[0099] When the first node receives the reply message from the second node, the first node obtains the sample retransmission times and the sample time difference between the time of sending the message and the time of replying the message.

[0100] The environment gives random fluctuations according to a certain strategy, simulating various scenarios in real network conditions, and randomly interfering with the message transmission of the first node. The first node sends a message to the second node according to the sample retransmission time. When the second node receives the message sent by the first node, it replies to the first node ACK (this process is not interfered with). When the first node receives the ACK message replied by the second node, it records the time t2 of receipt, and counts the total number of retransmissions cnt and the time difference Δt between message initiation and receipt.

[0101] In this embodiment, a message retransmission device based on reinforcement learning is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0102] This embodiment provides a message retransmission device based on reinforcement learning, such as Figure 3 As shown, including:

[0103] The collecting module 301 is used to collect the network status data of the node when receiving a message sending task; the network status data includes the signal-to-noise ratio, bandwidth utilization, packet loss rate and link quality of the current network.

[0104] The forming module 302 is used to form a state vector based on the network state data.

[0105] The calculation module 303 is used to obtain the target retransmission time through the reinforcement learning agent based on the state vector.

[0106] The sending module 304 is used to send a message to an external node based on the target retransmission time.

[0107] In some optional implementations, the calculation module 303 includes:

[0108] The preprocessing submodule is used to preprocess the state vector to obtain the target state vector; the preprocessing includes normalization and regularization operations.

[0109] The target retransmission time obtaining submodule is used to process the target state vector through a reinforcement learning agent to obtain the target retransmission time.

[0110] In some optional embodiments, the device further comprises:

[0111] The step number setting module is used to set the maximum number of training steps.

[0112] The training module is used to train the reinforcement learning agent in a preset network fluctuation environment when the current step number is not greater than the maximum training step number.

[0113] In some optional implementations, the training module includes:

[0114] The random number generation submodule is used to generate random numbers within the specified probability range and generate random exploration probabilities based on the current number of steps and the maximum number of training steps.

[0115] The sample retransmission time generation submodule is used to generate the sample retransmission time according to the size relationship between the random number and the random exploration probability.

[0116] The time difference obtaining submodule is used to perform data retransmission according to the sample retransmission time under a preset network fluctuation environment, and obtain the number of sample retransmissions and the sample time difference.

[0117] The update submodule is used to update the network parameters of the reinforcement learning agent according to the number of sample retransmissions and the sample time difference.

[0118] In some optional implementations, generating a sample retransmission time submodule includes:

[0119] A preset unit, used for randomly selecting a sample retransmission time within a preset interval by the reinforcement learning agent when the random number is less than the random exploration probability;

[0120] The exploration unit is used to obtain the sample retransmission time of the state vector corresponding to the environment of the preset network fluctuation of the reinforcement learning agent when the random number is greater than or equal to the random exploration probability.

[0121] In some optional implementations, obtaining the time difference submodule includes:

[0122] The message sending unit is used to set the sample retransmission time of the first node under a preset network fluctuation environment, and control the first node to send a message to the second node.

[0123] The data acquisition unit is used to acquire the sample retransmission times of the first node and the sample time difference between the time of sending the message and the time of replying the message when the first node receives the reply message from the second node.

[0124] The message retransmission device based on reinforcement learning in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0125] The embodiment of the present invention also provides a computer device having the above Figure 3 The message retransmission device based on reinforcement learning is shown.

[0126] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0127] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0128] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0129] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0131] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0132] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0133] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0134] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0135] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A message retransmission method based on reinforcement learning, characterized in that: The method comprises: When receiving a message sending task, collect network status data of this node; the network status data includes the signal-to-noise ratio, bandwidth utilization, packet loss rate and link quality of the current network; forming a state vector based on the network state data; Obtaining a target retransmission time through a reinforcement learning agent based on the state vector; A message is sent to an external node based on the target retransmission time.

2. The method according to claim 1, characterized in that: The step of obtaining the target retransmission time by using a reinforcement learning agent based on the state vector further includes: Preprocessing the state vector to obtain a target state vector; the preprocessing includes normalization and regularization operations; The target state vector is processed by the reinforcement learning agent to obtain the target retransmission time.

3. The method according to claim 1, characterized in that The method further comprises: Set the maximum number of training steps; When the current number of steps is not greater than the maximum number of training steps, the reinforcement learning agent is trained under a preset network fluctuation environment.

4. The method according to claim 3, characterized in that The step of training the reinforcement learning agent under a preset network fluctuation environment includes: Generate a random number within a specified probability range, and generate a random exploration probability according to the current step number and the maximum training step number; Generate a sample retransmission time according to the random number and the magnitude relationship of the random exploration probability; Under the preset network fluctuation environment, data is retransmitted according to the sample retransmission time to obtain the number of sample retransmissions and the sample time difference; The network parameters of the reinforcement learning agent are updated according to the number of sample retransmissions and the sample time difference.

5. The method according to claim 4, characterized in that The generating the sample retransmission time according to the random number and the magnitude relationship of the random exploration probability includes: When the random number is less than the random exploration probability, the reinforcement learning agent randomly selects the sample retransmission time within a preset interval; When the random number is greater than or equal to the random exploration probability, the sample retransmission time of the state vector of the reinforcement learning agent corresponding to the environment of preset network fluctuations is obtained.

6. The method according to claim 4, characterized in that The step of retransmitting data according to the sample retransmission time under the preset network fluctuation environment to obtain the number of sample retransmissions and the sample time difference includes: Under the preset network fluctuation environment, setting a sample retransmission time of the first node, and controlling the first node to send a message to the second node; When the first node receives the reply message from the second node, the first node obtains the sample retransmission times and the sample time difference between the time of sending the message and the time of replying the message.

7. A message retransmission device based on reinforcement learning, characterized in that: The device comprises: A collection module is used to collect network status data of the node when receiving a message sending task; the network status data includes the signal-to-noise ratio, bandwidth utilization, packet loss rate and link quality of the current network; A forming module, configured to form a state vector based on the network state data; A calculation module, configured to obtain a target retransmission time based on the state vector through a reinforcement learning agent; A sending module is used to send a message to an external node based on the target retransmission time.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the message retransmission method based on reinforcement learning according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the message retransmission method based on reinforcement learning according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the message retransmission method based on reinforcement learning according to any one of claims 1 to 6.

Images