Data delay retransmission method and related equipment

Abstract

The invention discloses a data delay retransmission method and related equipment, and the method comprises the steps: obtaining a current network packet loss state in response to a data retransmission request which is from external equipment and is used for indicating data transmission failure; determining a target data retransmission delay allocated for the current network packet loss state according to a target corresponding relationship between the data retransmission action and the network packet loss state; sending target data indicated by the data retransmission request to the external device at a data retransmission time point indicated by the target data retransmission delay; wherein the target corresponding relation between the data retransmission actions and the network packet loss states is obtained by performing reinforcement learning by taking the reward values corresponding to the data retransmission actions as rewards in the network packet loss states. According to the invention, data delay retransmission is carried out by applying a reinforcement learning means, so that the success rate of data retransmission can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data delayed retransmission method and related equipment. Background Art

[0002] In audio and video calls and live streaming services, voice quality is mainly affected by network packet loss. Packet loss occurs during the transmission process due to the instability of the transmission network, resulting in jamming and discontinuity of the sound, which brings a poor experience to the listener. There are many causes of network packet loss, such as network data congestion to a certain extent triggering the network device to actively discard packets, network equipment or transmission line failures causing data packet loss, wireless network interference causing data anomalies and packet discarding due to failure to pass verification, etc. Network packet loss is highly random and sudden, so it poses a great challenge to audio and video calls and live streaming services.

[0003] In the prior art, after receiving a data retransmission request, the electronic device will immediately send the lost data. However, under the mechanism of the prior art, the retransmitted data often continues to be lost due to network transmission problems, causing the party receiving the retransmitted data to continue to send requests to retransmit the data. Multiple data retransmissions not only consume transmission bandwidth resources, but also during multiple data retransmissions, the party receiving the retransmitted data needs to consume waiting time in order to wait for the data to arrive. If the number of retransmission failures is too many, the lost data may miss the effective packet receiving time and be abandoned. Summary of the invention

[0004] An embodiment of the present application provides a data delayed retransmission method and related equipment. The related equipment may include a data delayed retransmission device, an electronic device, a computer-readable storage medium and a computer program product, which can improve the success rate of data retransmission.

[0005] The embodiment of the present application provides a data delayed retransmission method, which is applied to a first electronic device, including:

[0006] In response to a data retransmission request from an external device indicating a data transmission failure, obtaining a current network packet loss status;

[0007] Determine a target data retransmission delay allocated for the current network packet loss state according to a target correspondence relationship between a data retransmission action and a network packet loss state;

[0008] At the data retransmission time point indicated by the target data retransmission delay, sending the target data indicated by the data retransmission request to the external device;

[0009] The target correspondence between the data retransmission action and the network packet loss state is obtained by using the reward values ​​corresponding to several data retransmission actions under several network packet loss states as rewards through reinforcement learning.

[0010] Accordingly, an embodiment of the present application provides a data delayed retransmission device, which is applied to a first electronic device, including:

[0011] A first acquisition unit, configured to acquire a current network packet loss state in response to a data retransmission request from an external device indicating a data transmission failure;

[0012] an allocating unit, configured to determine a target data retransmission delay allocated for the current network packet loss state according to a target correspondence relationship between a data retransmission action and a network packet loss state;

[0013] The first sending unit is used to send the target data indicated by the data retransmission request to the external device at the data retransmission time point indicated by the target data retransmission delay; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by using the reward values ​​corresponding to several data retransmission actions under several network packet loss states as rewards through reinforcement learning.

[0014] Optionally, in some embodiments of the present application, the data delay retransmission device may further include a first returning unit, a third acquiring unit, and an updating unit, as follows:

[0015] A first returning unit, configured to return to executing the step of responding to the data retransmission request from the external device for indicating data transmission failure, until the data retransmission request from the external device is no longer received;

[0016] The third acquisition unit is used to obtain the instant reward value corresponding to each network packet loss state in the current statistical period;

[0017] An updating unit is used to update the target correspondence based on the immediate reward value.

[0018] Optionally, in some embodiments of the present application, the third acquisition unit can be specifically used to obtain the number of first retransmission data corresponding to each network packet loss state in the current statistical period, and the number of successfully received second retransmission data fed back by the external device in response to the network packet loss state; based on the first number of retransmission data and the second number of retransmission data, determine the instant reward value corresponding to each network packet loss state in the current statistical period.

[0019] Optionally, in some embodiments of the present application, the allocation unit can be specifically used to determine the initial data retransmission delay allocated to the current network packet loss state and the initial reward value corresponding to the initial data retransmission delay under the current network packet loss state based on the target correspondence between the data retransmission action and the network packet loss state; based on the numerical value of the initial reward value, adjust the initial data retransmission delay to obtain the target data retransmission delay.

[0020] Optionally, in some embodiments of the present application, the data delay retransmission device may further include an initialization unit, a fourth acquisition unit, a determination unit, a third sending unit, a second return unit, a fifth acquisition unit, and an adjustment unit, as follows:

[0021] An initialization unit, used to initialize the initial correspondence between the data retransmission action and the network packet loss state when the service starts;

[0022] a fourth acquisition unit, configured to acquire a historical network packet loss status in response to a historical data retransmission request from the external device;

[0023] A determination unit, configured to determine a historical data retransmission delay allocated to the historical network packet loss state according to an initial correspondence between a data retransmission action and a network packet loss state;

[0024] A third sending unit, configured to send the historical target data indicated by the historical data retransmission request to the external device at the historical data retransmission time point indicated by the historical data retransmission delay;

[0025] A second returning unit, configured to return to executing the step of responding to the historical data retransmission request from the external device until the historical data retransmission request from the external device is no longer received;

[0026] A fifth acquisition unit is used to acquire a historical instant reward value corresponding to each historical network packet loss state within a number of historical statistical periods;

[0027] The adjusting unit is used to adjust the initial corresponding relationship multiple times based on the historical instant reward value until a preset stop condition is met to obtain the target corresponding relationship.

[0028] Optionally, in some embodiments of the present application, the determination unit can be specifically used to determine the probability of strategy selection corresponding to the historical network packet loss state; if it is determined to adopt a corresponding relationship strategy based on the strategy selection probability, the historical data retransmission delay assigned to the historical network packet loss state is determined according to the initial corresponding relationship between the data retransmission action and the network packet loss state; if it is determined to adopt a random strategy based on the strategy selection probability, the historical data retransmission delay is randomly assigned to the historical network packet loss state.

[0029] The embodiment of the present application provides a data delayed retransmission method, which is applied to a second electronic device, including:

[0030] In response to a data packet loss condition, obtaining a current network packet loss status;

[0031] The current network packet loss state is carried in a data retransmission request and sent to an external device, wherein the current network packet loss state is used by the external device to determine a target data retransmission delay allocated to the current network packet loss state according to a target correspondence between a data retransmission action and the network packet loss state, and to send the target data indicated by the data retransmission request at a data retransmission time point indicated by the target data retransmission delay;

[0032] receiving target data retransmitted by the external device;

[0033] The target correspondence between the data retransmission action and the network packet loss state is obtained by the external device performing reinforcement learning by using the reward values ​​corresponding to several data retransmission actions respectively under several network packet loss states as rewards.

[0034] Accordingly, an embodiment of the present application provides a data delayed retransmission device, which is applied to a second electronic device, including:

[0035] A second acquisition unit, configured to acquire a current network packet loss state in response to a data packet loss condition;

[0036] A second sending unit is used to carry the current network packet loss status in a data retransmission request and send it to an external device, wherein the current network packet loss status is used for the external device to determine a target data retransmission delay allocated to the current network packet loss status according to a target correspondence between a data retransmission action and the network packet loss status, and to send the target data indicated by the data retransmission request at a data retransmission time point indicated by the target data retransmission delay;

[0037] A receiving unit is used to receive the target data retransmitted by the external device; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by the external device performing reinforcement learning by using the reward values ​​corresponding to several data retransmission actions respectively under several network packet loss states as rewards.

[0038] Optionally, in some embodiments of the present application, the second acquisition unit can be specifically used to obtain the current network packet loss rate and the current network maximum continuous packet loss number in response to data packet loss conditions; and obtain the current network packet loss status based on the current network packet loss rate and the current network maximum continuous packet loss number.

[0039] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the steps in the data delayed retransmission method provided in the embodiment of the present application.

[0040] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps in the data delayed retransmission method provided in the embodiment of the present application are implemented.

[0041] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the data delayed retransmission method provided in the embodiment of the present application.

[0042] The embodiment of the present application provides a data delay retransmission method and related equipment, which can respond to a data retransmission request from an external device for indicating data transmission failure, obtain the current network packet loss state; determine the target data retransmission delay assigned to the current network packet loss state according to the target correspondence between the data retransmission action and the network packet loss state; at the data retransmission time point indicated by the target data retransmission delay, send the target data indicated by the data retransmission request to the external device; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by using the reward values ​​corresponding to several data retransmission actions respectively under several network packet loss states as rewards, and performing reinforcement learning. The present application can select the best time to perform data retransmission actions under different transmission network packet loss states by applying reinforcement learning to the network packet loss scenario, which not only reduces the bandwidth consumption caused by the data retransmission process, but also reduces the time consumption in the data retransmission process, thereby achieving the purpose of improving the success rate of data retransmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 It is a scenario diagram of the data delay retransmission method provided in an embodiment of the present application;

[0045] Figure 2 is a first flow chart of the data delay retransmission method provided in an embodiment of the present application;

[0046] Figure 3 is a second flow chart of the data delay retransmission method provided in an embodiment of the present application;

[0047] Figure 4 is a third flow chart of the data delay retransmission method provided in an embodiment of the present application;

[0048] Figure 5 It is a schematic diagram of reinforcement learning provided in an embodiment of the present application;

[0049] Figure 6 This is a schematic diagram of data retransmission interaction provided in an embodiment of the present application;

[0050] Figure 7 It is a first structural schematic diagram of a data delay retransmission device provided in an embodiment of the present application;

[0051] Figure 8 It is a second structural diagram of the data delayed retransmission device provided in an embodiment of the present application;

[0052] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0054] The embodiment of the present application provides a data delay retransmission method and related equipment, and the related equipment may include a data delay retransmission device, an electronic device, a computer-readable storage medium and a computer program product. The data delay retransmission device can be integrated in an electronic device, and the electronic device can be a terminal or a server.

[0055] It is understandable that the data delay retransmission method of this embodiment can be executed on the terminal, or on the server, or jointly by the terminal and the server. The above examples should not be understood as limiting the present application. Among them, the terminal may include a mobile phone, an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, an aircraft, a tablet computer, a laptop computer, or a personal computer (PC, Personal Computer), etc. A client may also be set on the terminal, and the client may be an application client or a browser client, etc. Among them, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The data delay retransmission method or device disclosed in this application, wherein multiple servers can be composed of a blockchain, and the server is a node on the blockchain.

[0056] like Figure 1 As shown, a first electronic device and a second electronic device jointly perform a data delay retransmission method as an example. The data delay retransmission system provided in the embodiment of the present application includes a first electronic device and a second electronic device, etc. The first electronic device and the second electronic device are connected via a network, such as a wired or wireless network connection.

[0057] Among them, the first electronic device can be used to: respond to a data retransmission request from the second electronic device for indicating a data transmission failure, obtain the current network packet loss state; determine the target data retransmission delay assigned to the current network packet loss state according to the target correspondence between the data retransmission action and the network packet loss state; at the data retransmission time point indicated by the target data retransmission delay, send the target data indicated by the data retransmission request to the second electronic device; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by reinforcement learning by using the reward values ​​corresponding to several data retransmission actions under several network packet loss states as rewards.

[0058] Among them, the second electronic device can be used to: obtain the current network packet loss state in response to the data packet loss condition; carry the current network packet loss state in the data retransmission request and send it to the first electronic device, the current network packet loss state is used for the first electronic device to determine the target data retransmission delay assigned to the current network packet loss state according to the target correspondence between the data retransmission action and the network packet loss state, and send the target data indicated by the data retransmission request at the data retransmission time point indicated by the target data retransmission delay; receive the target data retransmitted by the first electronic device; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by the first electronic device through reinforcement learning using the reward values ​​corresponding to several data retransmission actions respectively adopted under several network packet loss states as rewards.

[0059] Among them, Machine Learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by induction. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0060] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0061] This embodiment will be described from the perspective of a data delay retransmission device, which can be integrated in a first electronic device, which can be a server or a terminal. Figure 2 As shown, the specific process of the data delay retransmission method can be as follows:

[0062] 201. In response to a data retransmission request from an external device indicating a data transmission failure, obtain a current network packet loss status.

[0063] Among them, reinforcement learning is an important branch of machine learning. It is a machine learning method that uses environmental feedback as input, and realizes the mapping from environmental state to action through continuous exploration and self-learning. Reinforcement learning will obtain corresponding reward values ​​in each attempt of action, and optimize the action by maximizing the cumulative reward value. Reinforcement learning is different from supervised learning. Reinforcement learning does not require training samples in advance. It is an online learning technology. The reinforcement learning agent only needs to remember the state of its environment and the current strategy knowledge, and it can obtain the optimal decision suitable for the current environment from the accumulated exploration experience.

[0064] Among them, the external device refers to a device connected outside the current device. For example, the embodiment of the present application includes a first electronic device and a second electronic device. For the first electronic device, the second electronic device is the external device; for the second electronic device, the first electronic device is the external device.

[0065] Packet loss refers to the inability of one or more data packets to reach the destination through the network. After a packet loss event occurs, the lost data needs to be retransmitted to ensure the integrity of data transmission. In services such as audio and video calls and live broadcasts, voice quality is mainly affected by network packet loss. Due to the instability of the transmission network, packet loss occurs during the transmission process, causing the sound at the receiving end to freeze and become incoherent, which makes the listener experience very bad. There are many causes of network packet loss, such as network data congestion to a certain extent triggering the network device to actively discard packets, network equipment or transmission line failure causing packet loss, wireless network interference causing data anomalies and unable to pass verification and discard packets, etc. Network packet loss is highly random and sudden, so it poses a great challenge to audio and video calls and live broadcast services. Among them, the network packet loss status is the status that characterizes the data transmission loss in the network. Through the network packet loss status, we can understand the severity of network packet loss and make corresponding decisions.

[0066] There are many methods to resist network packet loss, including: FEC (forward error correction), PLC (Packet Loss Concealment), ARQ (Automatic Repeat Request), repeated transmission, etc. Among them, ARQ packet loss retransmission is an effective technology to solve packet loss. That is, when an electronic device detects that a data packet has timed out and has not been received, or finds that the received data packet is erroneous, it sends a request to an external device, requesting the external device to retransmit the erroneous data or multiple consecutive related data packets.

[0067] The data retransmission request is a request for requesting the first electronic device to retransmit data. The data retransmission request is a request for requesting the first electronic device to retransmit data after the second electronic device detects that a packet loss situation currently occurs.

[0068] For example, when the first electronic device receives a data retransmission request from the second electronic device, it can be known that data transmission fails in the current network, and the current network packet loss status carried in the data retransmission request can be obtained based on the received data retransmission request.

[0069] 202. Determine a target data retransmission delay allocated for the current network packet loss state according to a target correspondence relationship between a data retransmission action and a network packet loss state.

[0070] Among them, the target correspondence between the data retransmission action and the network packet loss state is the reward value corresponding to the use of several data retransmission actions respectively under several network packet loss states. The data retransmission success rate can be defined as a reward for reinforcement learning, and the reward value can be obtained from the reward for reinforcement learning. For example, in an embodiment of the present application, the target correspondence between the data retransmission action and the network packet loss state can be expressed in the form of a table with the size of the number of network packet loss states × the number of data retransmission actions. This table is used to record the reward value obtained by using a specific data retransmission action under a specific network packet loss state.

[0071] Among them, the data retransmission delay is the length of time required to wait for delayed data transmission. When packet loss occurs and data needs to be retransmitted, the data can be delayed for a certain period of time and then retransmitted. The length of the delay is called the data retransmission delay. For example, the data retransmission delay can be set to 10 milliseconds. Then, after receiving a data retransmission request, you can wait for 10 milliseconds before retransmitting the data.

[0072] For example, the first electronic device can allocate a target data retransmission delay for data retransmission to the current network packet loss status fed back by the second electronic device based on the target correspondence between the data retransmission action and the network packet loss status. That is, after receiving the data retransmission request, the data retransmission operation is not performed immediately, but is delayed for a target data retransmission delay period before performing the data retransmission operation.

[0073] Optionally, in one embodiment, the step of “determining a target data retransmission delay allocated for the current network packet loss state according to a target correspondence relationship between the data retransmission action and the network packet loss state” may include:

[0074] Determine, according to a target correspondence between a data retransmission action and a network packet loss state, an initial data retransmission delay allocated for the current network packet loss state, and an initial reward value corresponding to the initial data retransmission delay under the current network packet loss state;

[0075] Based on the value of the initial reward value, the initial data retransmission delay is adjusted to obtain the target data retransmission delay.

[0076] For example, in order to improve the adaptability between the selected target data retransmission delay and the current network packet loss state, the target data retransmission delay can also be selected through two steps of rough selection and fine adjustment. Since the embodiment of the present application will pre-set a number of data retransmission delays, such as 5 milliseconds, 10 milliseconds, 20 milliseconds, 30 milliseconds, 40 milliseconds, and 60 milliseconds. Then, according to the target correspondence between the data retransmission action and the network packet loss state, an initial data retransmission delay, such as 20 milliseconds, can be initially allocated to the current network packet loss state, and the reward value that can be obtained in the target correspondence relationship by using a data retransmission delay of 20 milliseconds in the current network packet loss state, and then according to the reward value, the initial data retransmission delay that has been obtained is fine-tuned, such as adjusting it to 19 milliseconds, or 21 milliseconds, so that the target data retransmission delay finally obtained is more adapted to the current network packet loss state.

[0077] 203. At the data retransmission time point indicated by the target data retransmission delay, send the target data indicated by the data retransmission request to the external device.

[0078] For example, after determining the time point for receiving the data retransmission request and the target data retransmission delay, the target data retransmission delay can be delayed at the time point for receiving the data retransmission request to obtain the data retransmission time point, and then the data retransmission operation can be performed at the data retransmission time point, wherein the target data indicated by the data retransmission request is the data that needs to be retransmitted.

[0079] Optionally, in one embodiment, after the step of “sending the target data indicated by the data retransmission request to the external device at the data retransmission time point indicated by the target data retransmission delay”, the following may also be included:

[0080] Returning to the step of responding to a data retransmission request from an external device for indicating a data transmission failure, until no more data retransmission requests from the external device are received;

[0081] Get the instant reward value corresponding to each network packet loss status in the current statistical period;

[0082] Update the target correspondence based on the immediate reward value.

[0083] For example, after executing the data retransmission operation, it is necessary to continue to receive data retransmission requests and continue the corresponding data retransmission operation until no more data retransmission requests are received. Since the target correspondence between the data retransmission action and the network packet loss state needs to be continuously maintained and updated in reinforcement learning, such as the target correspondence needs to be updated once within a statistical cycle, the instant reward value corresponding to each network packet loss state in the current statistical cycle can be obtained, and the target correspondence can be updated according to the instant reward value.

[0084] Optionally, in one embodiment, the step of “obtaining an instant reward value corresponding to each network packet loss state in the current statistical period” may include:

[0085] Acquire the number of first retransmitted data corresponding to each network packet loss state and the number of successfully received second retransmitted data fed back by an external device for the network packet loss state in a current statistical period;

[0086] Based on the first number of retransmitted data and the second number of retransmitted data, an instant reward value corresponding to each network packet loss state in the current statistical period is determined.

[0087] The first number of retransmitted data refers to the number of data retransmitted by the first electronic device to the second electronic device under a specific network packet loss state; the second number of retransmitted data refers to the number of data successfully received by the second electronic device. Since the second electronic device may not successfully receive all the data retransmitted from the first electronic device, the first number of retransmitted data and the second number of retransmitted data can be used to calculate the success rate of data retransmission.

[0088] For example, the first number of retransmitted data of data retransmitted by the first electronic device to the second electronic device under each network packet loss state in the current statistical period, and the second number of retransmitted data of successfully received data fed back by the second electronic device under each network packet loss state in the current statistical period can be obtained, and then, based on the first number of retransmitted data and the second number of retransmitted data, the instant reward value corresponding to each network packet loss state in the current statistical period is determined.

[0089] Among them, Figure 5As shown in the figure, reinforcement learning is an intelligent agent that continuously interacts with the environment to learn a mapping from environment to action. The goal of learning is to maximize the cumulative return. Reinforcement learning is also a kind of trial and error learning. It tries all possible actions in various environmental states, judges the quality of actions through the feedback (i.e. rewards) given by the environment, and finally obtains the mapping relationship between the environment and the optimal action. There are several elements involved in reinforcement learning: environmental state, action, and reward. The actual application of reinforcement learning requires defining these elements in combination with actual applications, and summarizing a complex reinforcement learning experience library through repeated verification or trial and error. This experience library is upgraded and enriched in the continuous self-learning process.

[0090] In one embodiment, for example, Figure 5 As shown, the technical solution of the present application is to use reinforcement learning to deal with the problem of network packet loss, trigger the operation of data delay retransmission based on the data retransmission request, and then affect the final data reception quality. As mentioned above, there are five core components in reinforcement learning, namely: environment (Environment), agent (Agent), state (State), action (Action) and reward (Reward). The reinforcement learning algorithm is the Q-Learning algorithm. In conjunction with the embodiments of the present application, the relevant elements of reinforcement learning are defined as follows:

[0091] 1. State

[0092] The network packet loss state can be determined as a state in reinforcement learning, wherein the network packet loss state is a state that characterizes the data transmission loss situation in the network. The network packet loss state can be used to understand the severity of the network packet loss and make corresponding decisions. For example, the second electronic device can determine the network packet loss state and feedback the network packet loss state to the first electronic device through a data retransmission request, so that the first electronic device can perform a data retransmission operation according to the network packet loss state.

[0093] 2. Action

[0094] The data delay retransmission operation can be determined as an action in reinforcement learning. The reinforcement learning system will give the configuration value action a of data delay retransmission according to the network packet loss status fed back by the second electronic device, that is, after the first electronic device receives the data retransmission request, it will delay a data retransmission delay time value before starting the data retransmission operation. The data retransmission delay time value can be preset. For example, the data retransmission delay time value can include 6 types: 5 milliseconds, 10 milliseconds, 20 milliseconds, 30 milliseconds, 40 milliseconds, and 60 milliseconds.

[0095] 3. Reward

[0096] The data retransmission success rate can be determined as a reward in reinforcement learning, and the data retransmission success rate can also be called an immediate reward value. For example, the number of first retransmission data of data retransmitted by the first electronic device to the second electronic device under a specific network packet loss state during the statistical period can be represented as send_num, and the number of second retransmission data successfully received by the second electronic device during the statistical period can be represented as rec_num. The calculation formula of the immediate reward value r can be as follows:

[0097]

[0098] 4. Q Form

[0099] In the reinforcement learning Q-learning algorithm, it is necessary to continuously maintain and update a Q table. The Q table is the target correspondence between the data retransmission action and the network packet loss state, which can be expressed in the form of a table with the size of the number of network packet loss states × the number of data retransmission actions. This table is used to record the reward value Q(s,a) obtained by taking a specific data retransmission action under a specific network packet loss state. The reward value in the Q table can be updated using the time difference method. Among them, this reward value is different from the above-mentioned immediate reward value r. The above-mentioned immediate reward value r represents a short-term reward, while the reward value Q(s,a) recorded in the Q table represents a long-term reward. The update formula of the reward value Q(s,a) can be as follows:

[0100]

[0101] Among them, α represents the learning rate (such as 0.01), γ represents the reward decay coefficient (such as 0.9), and s ′ Represents the state after state s takes the corresponding action, max a′ Q(s ′ ,a ′ ) represents the state s ′ The maximum reward you can get.

[0102] Optionally, in one embodiment, before the step of “responding to a data retransmission request from an external device for indicating a data transmission failure”, the following may also be included:

[0103] At the beginning of the service, the initial correspondence between the data retransmission action and the network packet loss status is initialized;

[0104] Responding to a historical data retransmission request from an external device, obtaining a historical network packet loss status;

[0105] Determine the historical data retransmission delay allocated to the historical network packet loss state according to the initial correspondence between the data retransmission action and the network packet loss state;

[0106] At the historical data retransmission time point indicated by the historical data retransmission delay, sending the historical target data indicated by the historical data retransmission request to the external device;

[0107] Returning to the step of responding to the historical data retransmission request from the external device until no historical data retransmission request from the external device is received;

[0108] Obtain the historical instant reward value corresponding to each historical network packet loss status within several historical statistical periods;

[0109] The initial correspondence is adjusted multiple times based on the historical instant reward value until the preset stop condition is met to obtain the target correspondence.

[0110] For example, an initial correspondence between data retransmission actions and network packet loss states, i.e., an initial Q table, can be created, and the number of network packet loss states is pre-set to 90 and the number of data retransmission actions is pre-set to 6. Then, the size of the initial Q table can be 90×6, and then an initial reward value is set for a specific data retransmission action under a specific network packet loss state based on existing experience. Then, a historical data retransmission request from a second electronic device is received, and the historical data retransmission request carries a historical network packet loss state. The first electronic device can determine the historical data retransmission delay assigned to the historical network packet loss state based on the initial correspondence between the data retransmission action and the network packet loss state, and at the historical data retransmission time point indicated by the historical data retransmission delay, send the historical target data indicated by the historical data retransmission request to the second electronic device, and then return to execute the step of responding to the historical data retransmission request from the second electronic device until no historical data retransmission request from the second electronic device is received. The historical instant reward value corresponding to each historical network packet loss state in a number of historical statistical cycles is obtained, and the Q table data is updated multiple times according to the historical instant reward value until the preset stop condition is met, and the target correspondence is obtained, i.e., the Q table is obtained.

[0111] Optionally, in one embodiment, the step of “determining, according to the initial correspondence between the data retransmission action and the network packet loss state, a historical data retransmission delay allocated to the historical network packet loss state” may include:

[0112] Determine the probability of strategy selection corresponding to the historical network packet loss status;

[0113] If the corresponding relationship strategy is adopted based on the strategy selection probability, the historical data retransmission delay allocated to the historical network packet loss state is determined according to the initial corresponding relationship between the data retransmission action and the network packet loss state;

[0114] If the random strategy is adopted based on the strategy selection probability, the historical data retransmission delay is randomly assigned to the historical network packet loss status.

[0115] Among them, since the selection of the data retransmission delay each time is mainly based on the size of the reward value in the corresponding relationship between the data retransmission action and the network packet loss state, but in the preliminary learning of reinforcement learning, the reward value recorded by the corresponding relationship between the data retransmission action and the network packet loss state is not accurate enough. If the data retransmission delay is selected according to the corresponding relationship between the data retransmission action and the network packet loss state at this time, it will cause a large error. In addition, after a period of reinforcement learning, the reinforcement learning agent may have a relatively fixed action selection and cannot effectively explore the environment. Therefore, when selecting the data retransmission delay, the embodiment of the present application can not completely follow the corresponding relationship between the data retransmission action and the network packet loss state, but select the probability by setting the strategy, so that there is a part of the probability that the data retransmission delay can be randomly selected.

[0116] Among them, the strategy selection probability is the probability of indicating which strategy is currently selected, and the strategy selection probability can be set according to the actual situation. For example, in the embodiment of the present application, there are two strategies for network packet loss: the first is to allocate data retransmission delay for network packet loss state according to the correspondence between data retransmission action and network packet loss state; the second is to randomly allocate data retransmission delay for network packet loss state. The strategy selection probability can be used to determine which strategy to use.

[0117] For example, the strategy selection probability corresponding to the historical network packet loss state can be determined. If the corresponding relationship strategy is adopted based on the strategy selection probability, that is, the strategy selection probability at this time satisfies the situation of selecting an action based on the correspondence between the data retransmission action and the network packet loss state, then the historical data retransmission delay assigned to the historical network packet loss state can be determined based on the initial correspondence between the data retransmission action and the network packet loss state; if the random strategy is adopted based on the strategy selection probability, that is, the strategy selection probability at this time satisfies the situation of randomly selecting an action, then the historical data retransmission delay is randomly assigned to the historical network packet loss state.

[0118] In addition, in one embodiment, since the reward value recorded by the correspondence between the data retransmission action and the network packet loss state is not accurate enough in the initial learning of reinforcement learning, the strategy selection probability value corresponding to the random strategy can be set relatively large, and then the strategy selection probability value corresponding to the random strategy can be gradually reduced. For example, it can be set to decide whether to adjust the strategy selection probability after a certain length of time or a certain number of rounds of reinforcement learning; for another example, it can also be decided whether to adjust the strategy selection probability based on the sparseness of the reward value greater than the preset value in the correspondence between the data retransmission action and the network packet loss state.

[0119] In the prior art, after receiving a data retransmission request, the electronic device will immediately send the lost data packet to the external device. However, not all retransmitted data packets can successfully reach the external device. When the transmission network quality is poor and there are many continuous packet losses, the retransmitted data packet is likely to fail to be sent successfully. At this time, a second data retransmission request will be initiated. After receiving the second data retransmission request, the electronic device will trigger the second data retransmission. If data is still lost during this data retransmission, a third data retransmission request will be initiated. This will be repeated until the external device successfully receives the lost data packet, or the number of data retransmissions reaches the set upper limit of the number of data retransmissions.

[0120] However, during each data retransmission process, there is a delay in data transmission, and the specific value of the delay is not stable. Its value is affected by the transmission network conditions. Therefore, the party receiving the retransmitted data needs to wait until the data packet is received, or the waiting time exceeds the preset time interval (at this time, the packet reception is determined to have failed). If the transmission network quality is poor, it is very likely that multiple consecutive data retransmission failures will occur. This situation not only consumes the bandwidth consumption of invalid round-trip transmission data, but also wastes the waiting time of the party receiving the retransmitted data, causing it to lose the important window period for recovering the lost packet data.

[0121] Therefore, the technical solution of the present application proposes a delayed sending mechanism of automatic retransmission request (ARQ) based on reinforcement learning, which can be applied to various application fields such as live video, games, audio broadcasting, etc. The technical solution of the present application can apply reinforcement learning to trigger the data delay retransmission action based on the current network packet loss status, thereby affecting the final data reception quality. The technical solution of the present application selects the best time to perform data delay retransmission action under different transmission network packet loss states through reinforcement learning, thereby improving the success rate of data retransmission in one time, that is, reducing bandwidth consumption and time consumption.

[0122] As can be seen from the above, the present application can obtain the current network packet loss state in response to a data retransmission request from an external device for indicating data transmission failure; determine the target data retransmission delay assigned to the current network packet loss state according to the target correspondence between the data retransmission action and the network packet loss state; send the target data indicated by the data retransmission request to the external device at the data retransmission time point indicated by the target data retransmission delay; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by reinforcement learning by using the reward values ​​corresponding to several data retransmission actions under several network packet loss states as rewards. The present application can select the best time to perform data retransmission actions under different transmission network packet loss states by applying reinforcement learning to the network packet loss scenario, which not only reduces the bandwidth consumption caused by the data retransmission process, but also reduces the time consumption in the data retransmission process, thereby achieving the purpose of improving the success rate of data retransmission.

[0123] This embodiment will be described from the perspective of a data delay retransmission device, which can be integrated in a second electronic device, which can be a server or a terminal. Figure 3 As shown, the specific process of the data delay retransmission method can be as follows:

[0124] 301. In response to a data packet loss condition, obtain a current network packet loss status.

[0125] The data packet loss condition may be a condition indicating that data packet loss occurs in the current network. When the second electronic device detects that data packet loss occurs in the current network, it is considered that the data packet loss condition is met.

[0126] For example, when the second electronic device detects that data packet loss occurs in the current network, it can be considered that the data packet loss condition is met, and the second electronic device can obtain the current network packet loss status.

[0127] Optionally, in one embodiment, the step of “obtaining the current network packet loss status in response to the data packet loss condition” may include:

[0128] In response to the data packet loss condition, obtain the current network packet loss rate and the current network maximum number of consecutive packet losses;

[0129] Get the current network packet loss status based on the current network packet loss rate and the current network maximum number of consecutive packet losses.

[0130] For example, since the current network packet loss rate and the current network's maximum number of consecutive packet losses can best reflect the current transmission network's packet loss characteristics, the current network packet loss rate and the current network's maximum number of consecutive packet losses can be used to define the state of reinforcement learning. Then, when the second electronic device detects data packet loss in the current network, it can be considered that the data packet loss condition is met, and the second electronic device can obtain the current network packet loss rate and the current network's maximum number of consecutive packet losses, and obtain the current network packet loss state based on the current network packet loss rate and the current network's maximum number of consecutive packet losses.

[0131] In one embodiment, the second electronic device can perform packet loss statistics on the data of the N frames before the current moment, that is, based on the timestamp information recorded in each frame of the data packet, where the timestamp information increases with time, it is determined whether the received adjacent data packets meet the normal timestamp spacing. If the normal timestamp spacing is not met, it means that there is a packet loss situation. At this time, the number of lost data packets in a statistical window can be counted to obtain the packet loss rate lossrate. In order to quantify the current network packet loss rate, the following 9 current network packet loss rate states State_lossrate can be defined:

[0132] If lossrate<3then State_lossrate=0

[0133] If 3≤lossrate<5then State_lossrate=1

[0134] If 5≤lossrate<10then State_lossrate=2

[0135] If 10≤lossrate<15then State_lossrate=3

[0136] If 15≤lossrate<20then State_lossrate=4

[0137] If 20≤lossrate<25then State_lossrate=5

[0138] If 25≤lossrate<30then State_lossrate=6

[0139] If 30≤lossrate<50then State_lossrate=7

[0140] Else State_lossrate = 8

[0141] In one embodiment, the second electronic device may determine the current network maximum continuous packet loss number State_contlost based on the maximum number of continuous packet losses in the timestamp packet loss judgment statistical window. In order to quantify the current network maximum continuous packet loss number, the following 10 maximum continuous packet loss states may be defined:

[0142] If contlost<2then State_contlost=0

[0143] If 2≤contlost<3then State_contlost=1

[0144] If 3≤contlost<4 then State_contlost=2

[0145] If 4≤contlost<5 then State_contlost=3

[0146] If 5≤contlost<6 then State_contlost=4

[0147] If 6≤contlost<8 then State_contlost=5

[0148] If 8≤contlost<10 then State_contlost=6

[0149] If 10≤contlost<15 then State_contlost=7

[0150] If 15≤contlost<20 then State_contlost=8

[0151] Else State_contlost = 9

[0152] According to the above current network packet loss rate and the current maximum number of consecutive network packet losses, a total of 90 current network packet loss states can be formed. Therefore, the state value St of reinforcement learning can be defined as follows:

[0153] St=10×State_lossrate+State_contlost

[0154] 302. Carry the current network packet loss status in the data retransmission request and send it to the external device. The current network packet loss status is used for the external device to determine the target data retransmission delay assigned to the current network packet loss status according to the target correspondence between the data retransmission action and the network packet loss status, and send the target data indicated by the data retransmission request at the data retransmission time point indicated by the target data retransmission delay.

[0155] For example, the second electronic device can carry the current network packet loss status in a data retransmission request and send it to the first electronic device, so that the first electronic device can assign a target data retransmission delay to the current network packet loss status based on the target correspondence between the data retransmission action and the network packet loss status, and send the target data indicated by the data retransmission request to the second electronic device at the data retransmission time point indicated by the target data retransmission delay.

[0156] 303. Receive target data retransmitted by an external device.

[0157] For example, the second electronic device can receive the target data retransmitted by the first electronic device.

[0158] In the prior art, after receiving a data retransmission request, an electronic device will immediately send the lost target data to an external device. However, the technical solution of the prior art usually results in the retransmitted target data being lost again due to network transmission problems, resulting in the data retransmission request being sent again. Multiple data retransmissions not only consume transmission bandwidth resources, but also during multiple retransmissions, the party receiving the retransmitted data needs to spend waiting time waiting for the data to arrive. As the number of retransmission failures increases, the time spent on data retransmission also increases, which may eventually cause the lost data packet to miss the effective packet receiving time window and fail to receive.

[0159] The technical solution of the present application proposes a delayed sending mechanism of automatic retransmission request (ARQ) based on reinforcement learning, which can be applied to various application fields such as live video, games, audio broadcasting, etc. The technical solution of the present application can apply reinforcement learning to trigger the data delay retransmission action based on the current network packet loss status, thereby affecting the final data reception quality. The technical solution of the present application selects the best time to perform data delay retransmission action under different transmission network packet loss states through reinforcement learning, thereby improving the success rate of data retransmission in one time, that is, reducing bandwidth consumption and time consumption.

[0160] As can be seen from the above, this embodiment can obtain the current network packet loss state in response to a data retransmission request from an external device for indicating data transmission failure; determine the target data retransmission delay assigned to the current network packet loss state according to the target correspondence between the data retransmission action and the network packet loss state; send the target data indicated by the data retransmission request to the external device at the data retransmission time point indicated by the target data retransmission delay; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by using the reward values ​​corresponding to several data retransmission actions respectively under several network packet loss states as rewards through reinforcement learning. This application can select the best time to perform data retransmission actions under different transmission network packet loss states by applying reinforcement learning to network packet loss scenarios, thereby reducing both the bandwidth consumption caused by the data retransmission process and the time consumption in the data retransmission process, thereby achieving the purpose of improving the success rate of data retransmission.

[0161] According to the method described in the previous embodiment, the data delay retransmission device is specifically integrated in the first electronic device and the second electronic device as an example for further detailed description. The present application embodiment provides a data delay retransmission method, such as Figure 4 As shown, the specific process of the data delay retransmission method can be as follows:

[0162] 401. The first electronic device initializes the Q table.

[0163] For example, in the reinforcement learning Q-learning algorithm, a Q table needs to be continuously maintained and updated. The Q table is a table of the number of network packet loss states × the number of data retransmission actions. This table is used to record the long-term reward value Q(s,a) corresponding to the corresponding action taken in each state.

[0164] 402. When a packet loss situation is detected, the second electronic device sends a data retransmission request to the first electronic device. The data retransmission request carries the current network packet loss status.

[0165] For example, Figure 6 As shown, when a packet loss is detected, the second electronic device sends a data retransmission request to the first electronic device, and the data retransmission request carries the current network packet loss status, wherein the current network packet loss status includes the current network packet loss rate and the current network maximum number of consecutive packet losses.

[0166] In one embodiment, the second electronic device can perform packet loss statistics on the data of the N frames before the current moment, that is, based on the timestamp information recorded in each frame of the data packet, where the timestamp information increases with time, it is determined whether the received adjacent data packets meet the normal timestamp interval. If the normal timestamp interval is not met, it means that there is a packet loss situation. At this time, the number of lost data packets in a statistical window can be counted to obtain the current network packet loss rate lossrate. In order to quantify the current network packet loss rate, the following 9 packet loss rate states can be defined:

[0167] If lossrate<3then State_lossrate=0

[0168] If 3≤lossrate<5then State_lossrate=1

[0169] If 5≤lossrate<10then State_lossrate=2

[0170] If 10≤lossrate<15then State_lossrate=3

[0171] If 15≤lossrate<20then State_lossrate=4

[0172] If 20≤lossrate<25then State_lossrate=5

[0173] If 25≤lossrate<30then State_lossrate=6

[0174] If 30≤lossrate<50then State_lossrate=7

[0175] Else State_lossrate = 8

[0176] In one embodiment, the second electronic device can determine the maximum number of consecutive packet losses in the statistical window based on the timestamp packet loss, that is, the current maximum number of consecutive packet losses in the network. In order to quantify the current maximum number of consecutive packet losses in the network, the following 10 maximum consecutive packet loss states can be defined:

[0177] If contlost<2then State_contlost=0

[0178] If 2≤contlost<3then State_contlost=1

[0179] If 3≤contlost<4then State_contlost=2

[0180] If 4≤contlost<5then State_contlost=3

[0181] If 5≤contlost<6then State_contlost=4

[0182] If 6≤contlost<8then State_contlost=5

[0183] If 8≤contlost<10then State_contlost=6

[0184] If 10≤contlost<15then State_contlost=7

[0185] If 15≤contlost<20then State_contlost=8

[0186] Else State_contlost = 9

[0187] According to the above current network packet loss rate and the current maximum number of consecutive network packet losses, a total of 90 current network packet loss states can be formed. Therefore, the state value St of reinforcement learning can be defined as follows:

[0188] St=10×State_lossrate+State_contlost

[0189] 403. The first electronic device allocates a data retransmission delay for the current network packet loss status.

[0190] For example, multiple data retransmission delay values ​​can be pre-set: 5 milliseconds, 10 milliseconds, 20 milliseconds, 30 milliseconds, 40 milliseconds, and 60 milliseconds. The first electronic device can allocate a data retransmission delay that matches the current network packet loss status based on feedback from the second electronic device.

[0191] Among them, the first electronic device can allocate data retransmission delay for the current network packet loss state based on the Q table. Since in the Q table, the current network packet loss state will correspond to multiple data retransmission delays, and each corresponds to a different reward value, the data retransmission delay corresponding to a relatively large reward value can be allocated to the current network packet loss state.

[0192] In addition, since the reward value recorded by the correspondence between the data retransmission action and the network packet loss state in the initial learning of reinforcement learning is not accurate enough, if the data retransmission delay is selected according to the correspondence between the data retransmission action and the network packet loss state, it will cause a large error. Therefore, when selecting the data retransmission delay, the embodiment of the present application may not completely follow the correspondence between the data retransmission action and the network packet loss state, but select the probability by setting the strategy, so that there is a part of the probability that the data retransmission delay can be randomly selected.

[0193] For example, the strategy selection probability corresponding to the current network packet loss state can be determined. If the corresponding relationship strategy is adopted based on the strategy selection probability, that is, the strategy selection probability at this time satisfies the situation of selecting an action using the Q table, then the data retransmission delay allocated to the current network packet loss state can be determined according to the Q table; if the random strategy is adopted based on the strategy selection probability, that is, the strategy selection probability at this time satisfies the situation of randomly selecting an action, then the historical data retransmission delay is randomly allocated to the current network packet loss state.

[0194] 404. The first electronic device sends the target data indicated by the data retransmission request to the second electronic device at the data retransmission time point indicated by the data retransmission delay.

[0195] For example, the first electronic device may wait for the data retransmission delay time, i.e., the data retransmission time point, at the time point of receiving the data retransmission request, and send the target data indicated by the data retransmission request to the second electronic device at the data retransmission time point.

[0196] 405. The second electronic device receives target data from the first electronic device.

[0197] 406. The first electronic device obtains an instant reward value corresponding to each network packet loss state within a statistical period, and updates the Q table based on the instant reward value.

[0198] For example, the data retransmission success rate can be used as an instant reward value, that is, the first electronic device can obtain the number of data sent by the first electronic device corresponding to each network packet loss state within the statistical period, send_num, and the number of data successfully received by the second electronic device rec_num, and calculate the instant reward value r accordingly.

[0199]

[0200] In addition, since a Q table needs to be continuously maintained and updated in the reinforcement learning Q-learning algorithm, the first electronic device can update the Q table based on the immediate reward value. The reward value recorded in the Q table is different from the immediate reward value r, which represents a short-term reward, while the reward value Q(s,a) recorded in the Q table represents a long-term reward. The update formula of the reward value Q(s,a) can be as follows:

[0201]

[0202] Among them, α represents the learning rate (such as 0.01), γ represents the reward decay coefficient (such as 0.9), s′ represents the state after the state s takes the corresponding action, and max a′ Q(s′,a′) represents the maximum reward that can be obtained in state s′.

[0203] As can be seen from the above, in this embodiment, the Q table can be initialized by the first electronic device; when a packet loss situation is detected, a data retransmission request is sent to the first electronic device by the second electronic device, and the data retransmission request carries the current network packet loss status; the first electronic device allocates a data retransmission delay for the current network packet loss status; the first electronic device sends the target data indicated by the data retransmission request to the second electronic device at the data retransmission time point indicated by the data retransmission delay; the second electronic device receives the target data from the first electronic device; the first electronic device obtains the instant reward value corresponding to each network packet loss status within the statistical period, and updates the Q table based on the instant reward value. This application can apply reinforcement learning to the network packet loss scenario, so that the best time can be selected for data retransmission under different transmission network packet loss states, which not only reduces the bandwidth consumption caused by the data retransmission process, but also reduces the time consumption in the data retransmission process, thereby achieving the purpose of improving the success rate of data retransmission.

[0204] In order to better implement the above method, the embodiment of the present application also provides a data delay retransmission device, which is applied to a first electronic device, such as Figure 7 As shown, the data delay retransmission device may include a first acquisition unit 701, an allocation unit 702, and a first sending unit 703, as follows:

[0205] A first acquisition unit 701 is used to obtain a current network packet loss state in response to a data retransmission request from an external device indicating a data transmission failure;

[0206] An allocating unit 702 is used to determine a target data retransmission delay allocated for the current network packet loss state according to a target correspondence relationship between a data retransmission action and a network packet loss state;

[0207] The first sending unit 703 is used to send the target data indicated by the data retransmission request to the external device at the data retransmission time point indicated by the target data retransmission delay; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by using the reward values ​​corresponding to several data retransmission actions under several network packet loss states as rewards through reinforcement learning.

[0208] Optionally, in some embodiments of the present application, the data delay retransmission device may further include a first returning unit, a third acquiring unit, and an updating unit, as follows:

[0209] A first returning unit, configured to return to executing the step of responding to the data retransmission request from the external device for indicating data transmission failure, until the data retransmission request from the external device is no longer received;

[0210] The third acquisition unit is used to obtain the instant reward value corresponding to each network packet loss state in the current statistical period;

[0211] An updating unit is used to update the target correspondence based on the immediate reward value.

[0212] Optionally, in some embodiments of the present application, the third acquisition unit can be specifically used to obtain the number of first retransmission data corresponding to each network packet loss state in the current statistical period, and the number of successfully received second retransmission data fed back by the external device in response to the network packet loss state; based on the first number of retransmission data and the second number of retransmission data, determine the instant reward value corresponding to each network packet loss state in the current statistical period.

[0213] Optionally, in some embodiments of the present application, the allocation unit can be specifically used to determine the initial data retransmission delay allocated to the current network packet loss state and the initial reward value corresponding to the initial data retransmission delay under the current network packet loss state based on the target correspondence between the data retransmission action and the network packet loss state; based on the numerical value of the initial reward value, adjust the initial data retransmission delay to obtain the target data retransmission delay.

[0214] Optionally, in some embodiments of the present application, the data delay retransmission device may further include an initialization unit, a fourth acquisition unit, a determination unit, a third sending unit, a second return unit, a fifth acquisition unit, and an adjustment unit, as follows:

[0215] An initialization unit, used to initialize the initial correspondence between the data retransmission action and the network packet loss state when the service starts;

[0216] a fourth acquiring unit, configured to acquire a historical network packet loss status in response to a historical data retransmission request from the external device;

[0217] A determination unit, configured to determine a historical data retransmission delay allocated to the historical network packet loss state according to an initial correspondence between a data retransmission action and a network packet loss state;

[0218] A third sending unit, configured to send the historical target data indicated by the historical data retransmission request to the external device at the historical data retransmission time point indicated by the historical data retransmission delay;

[0219] A second returning unit, configured to return to executing the step of responding to the historical data retransmission request from the external device until the historical data retransmission request from the external device is no longer received;

[0220] A fifth acquisition unit is used to acquire a historical instant reward value corresponding to each historical network packet loss state within a number of historical statistical periods;

[0221] The adjustment unit is used to adjust the initial corresponding relationship multiple times based on the historical instant reward value until a preset stop condition is met to obtain the target corresponding relationship.

[0222] Optionally, in some embodiments of the present application, the determination unit can be specifically used to determine the probability of strategy selection corresponding to the historical network packet loss state; if it is determined to adopt a corresponding relationship strategy based on the strategy selection probability, the historical data retransmission delay assigned to the historical network packet loss state is determined according to the initial corresponding relationship between the data retransmission action and the network packet loss state; if it is determined to adopt a random strategy based on the strategy selection probability, the historical data retransmission delay is randomly assigned to the historical network packet loss state.

[0223] In order to better implement the above method, the embodiment of the present application also provides a data delay retransmission device, which is applied to a second electronic device, such as Figure 8 As shown, the data delay retransmission device may include a second acquisition unit 801, a second sending unit 802, and a receiving unit 803, as follows:

[0224] The second acquisition unit 801 is used to obtain the current network packet loss state in response to the data packet loss condition;

[0225] The second sending unit 802 is used to carry the current network packet loss state in the data retransmission request and send it to the external device, where the current network packet loss state is used for the external device to determine the target data retransmission delay allocated for the current network packet loss state according to the target correspondence between the data retransmission action and the network packet loss state, and send the target data indicated by the data retransmission request at the data retransmission time point indicated by the target data retransmission delay;

[0226] The receiving unit 803 is used to receive the target data retransmitted by the external device; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by the external device performing reinforcement learning by using the reward values ​​corresponding to several data retransmission actions respectively under several network packet loss states as rewards.

[0227] Optionally, in some embodiments of the present application, the second acquisition unit can be specifically used to obtain the current network packet loss rate and the current network maximum continuous packet loss number in response to data packet loss conditions; and obtain the current network packet loss status based on the current network packet loss rate and the current network maximum continuous packet loss number.

[0228] As can be seen from the above, in this embodiment, the current network packet loss state can be obtained by the first acquisition unit 701 in response to the data retransmission request from the external device for indicating the failure of data transmission; the target data retransmission delay assigned to the current network packet loss state can be determined by the allocation unit 702 according to the target correspondence between the data retransmission action and the network packet loss state; the target data indicated by the data retransmission request is sent to the external device by the first sending unit 703 at the data retransmission time point indicated by the target data retransmission delay; wherein, the target correspondence between the data retransmission action and the network packet loss state is obtained by using the reward values ​​corresponding to several data retransmission actions respectively under several network packet loss states as rewards, and performing reinforcement learning. The present application can select the best time to perform data retransmission actions under different transmission network packet loss states by applying reinforcement learning to the network packet loss scenario, which not only reduces the bandwidth consumption caused by the data retransmission process, but also reduces the time consumption in the data retransmission process, thereby achieving the purpose of improving the success rate of data retransmission.

[0229] The present application also provides an electronic device, such as Fig. 9 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present application, and the electronic device may be a terminal or a server, etc. Specifically:

[0230] The electronic device may include components such as a processor 901 with one or more processing cores, a memory 902 with one or more computer-readable storage media, a power supply 903, and an input unit 904. Those skilled in the art will appreciate that Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0231] The processor 901 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 902, and calling data stored in the memory 902, the processor 901 performs various functions of the electronic device and processes data. Optionally, the processor 901 may include one or more processing cores; preferably, the processor 901 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 901.

[0232] The memory 902 can be used to store software programs and modules. The processor 901 executes various functional applications and data processing by running the software programs and modules stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 902 may also include a memory controller to provide the processor 901 with access to the memory 902.

[0233] The electronic device also includes a power supply 903 for supplying power to each component. Preferably, the power supply 903 can be logically connected to the processor 901 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 903 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0234] The electronic device may further include an input unit 904, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0235] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 901 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 902 according to the following instructions, and the processor 901 will run the application programs stored in the memory 902, thereby realizing various functions, as follows:

[0236] In response to a data retransmission request from an external device for indicating a data transmission failure, a current network packet loss state is obtained; based on a target correspondence between a data retransmission action and the network packet loss state, a target data retransmission delay assigned to the current network packet loss state is determined; at a data retransmission time point indicated by the target data retransmission delay, the target data indicated by the data retransmission request is sent to the external device; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by reinforcement learning using reward values ​​corresponding to several data retransmission actions under several network packet loss states as rewards.

[0237] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0238] As can be seen from the above, this embodiment can obtain the current network packet loss state in response to a data retransmission request from an external device for indicating data transmission failure; determine the target data retransmission delay assigned to the current network packet loss state according to the target correspondence between the data retransmission action and the network packet loss state; send the target data indicated by the data retransmission request to the external device at the data retransmission time point indicated by the target data retransmission delay; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by using the reward values ​​corresponding to several data retransmission actions respectively under several network packet loss states as rewards through reinforcement learning. This application can select the best time to perform data retransmission actions under different transmission network packet loss states by applying reinforcement learning to network packet loss scenarios, thereby reducing both the bandwidth consumption caused by the data retransmission process and the time consumption in the data retransmission process, thereby achieving the purpose of improving the success rate of data retransmission.

[0239] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0240] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the data delay retransmission methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0241] In response to a data retransmission request from an external device for indicating a data transmission failure, a current network packet loss state is obtained; based on a target correspondence between a data retransmission action and the network packet loss state, a target data retransmission delay assigned to the current network packet loss state is determined; at a data retransmission time point indicated by the target data retransmission delay, the target data indicated by the data retransmission request is sent to the external device; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by reinforcement learning using reward values ​​corresponding to several data retransmission actions under several network packet loss states as rewards.

[0242] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0243] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0244] Since the instructions stored in the computer-readable storage medium can execute the steps in any one of the data delayed retransmission methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the data delayed retransmission methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0245] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations of the above-mentioned data delay retransmission aspect.

[0246] The above is a detailed introduction to a data delayed retransmission method and related equipment provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data delayed retransmission method, characterized in that: Applicable to a first electronic device, comprising: In response to a data retransmission request from an external device indicating a data transmission failure, obtaining a current network packet loss status; Determining a target data retransmission delay allocated for the current network packet loss state according to a target correspondence relationship between a data retransmission action and a network packet loss state; At the data retransmission time point indicated by the target data retransmission delay, sending the target data indicated by the data retransmission request to the external device; The target correspondence between the data retransmission action and the network packet loss state is obtained by using the reward values ​​corresponding to several data retransmission actions under several network packet loss states as rewards through reinforcement learning.

2. The data delayed retransmission method according to claim 1, characterized in that: After sending the target data indicated by the data retransmission request to the external device at the data retransmission time point indicated by the target data retransmission delay, the method further includes: Returning to the step of responding to the data retransmission request from the external device indicating data transmission failure, until the data retransmission request from the external device is no longer received; Get the instant reward value corresponding to each network packet loss status in the current statistical period; The target correspondence is updated based on the immediate reward value.

3. The data delayed retransmission method according to claim 2, characterized in that: The obtaining of the instant reward value corresponding to each network packet loss state in the current statistical period includes: Acquire the number of first retransmitted data corresponding to each network packet loss state and the number of successfully received second retransmitted data fed back by the external device for the network packet loss state in the current statistical period; Based on the first number of retransmitted data and the second number of retransmitted data, an instant reward value corresponding to each network packet loss state in a current statistical period is determined.

4. The data delayed retransmission method according to claim 1, characterized in that: The determining, according to the target correspondence between the data retransmission action and the network packet loss state, a target data retransmission delay allocated for the current network packet loss state comprises: Determine, according to a target correspondence between a data retransmission action and a network packet loss state, an initial data retransmission delay allocated for the current network packet loss state, and an initial reward value corresponding to the initial data retransmission delay under the current network packet loss state; Based on the numerical value of the initial reward value, the initial data retransmission delay is adjusted to obtain a target data retransmission delay.

5. The data delayed retransmission method according to claim 1, characterized in that: Before responding to a data retransmission request from an external device indicating a data transmission failure, the method further includes: At the beginning of the service, the initial correspondence between the data retransmission action and the network packet loss status is initialized; In response to a historical data retransmission request from the external device, obtaining a historical network packet loss status; Determine, according to an initial correspondence between a data retransmission action and a network packet loss state, a historical data retransmission delay allocated to the historical network packet loss state; At the historical data retransmission time point indicated by the historical data retransmission delay, sending the historical target data indicated by the historical data retransmission request to the external device; Returning to the step of responding to the historical data retransmission request from the external device until the historical data retransmission request from the external device is no longer received; Obtain the historical instant reward value corresponding to each historical network packet loss status within several historical statistical periods; The initial corresponding relationship is adjusted multiple times based on the historical instant reward value until a preset stop condition is met to obtain the target corresponding relationship.

6. The data delayed retransmission method according to claim 5, characterized in that: The determining, according to the initial correspondence between the data retransmission action and the network packet loss state, a historical data retransmission delay allocated to the historical network packet loss state comprises: Determine the probability of strategy selection corresponding to the historical network packet loss state; If the corresponding relationship strategy is adopted based on the strategy selection probability, the historical data retransmission delay allocated to the historical network packet loss state is determined according to the initial corresponding relationship between the data retransmission action and the network packet loss state; If it is determined to adopt a random strategy based on the strategy selection probability, a historical data retransmission delay is randomly assigned to the historical network packet loss state.

7. A data delayed retransmission method, characterized in that: Applicable to a second electronic device, comprising: In response to a data packet loss condition, obtaining a current network packet loss status; The current network packet loss state is carried in a data retransmission request and sent to an external device, wherein the current network packet loss state is used by the external device to determine a target data retransmission delay allocated to the current network packet loss state according to a target correspondence between a data retransmission action and the network packet loss state, and to send the target data indicated by the data retransmission request at a data retransmission time point indicated by the target data retransmission delay; receiving target data retransmitted by the external device; The target correspondence between the data retransmission action and the network packet loss state is obtained by the external device performing reinforcement learning by using the reward values ​​corresponding to several data retransmission actions respectively under several network packet loss states as rewards.

8. The data delayed retransmission method according to claim 7, characterized in that: The step of obtaining the current network packet loss status in response to the data packet loss condition includes: In response to the data packet loss condition, obtain the current network packet loss rate and the current network maximum number of consecutive packet losses; Based on the current network packet loss rate and the current network maximum continuous packet loss number, the current network packet loss state is obtained.

9. A data delayed retransmission device, characterized in that: Applicable to a first electronic device, comprising: A first acquisition unit, configured to acquire a current network packet loss state in response to a data retransmission request from an external device indicating a data transmission failure; an allocating unit, configured to determine a target data retransmission delay allocated for the current network packet loss state according to a target correspondence relationship between a data retransmission action and a network packet loss state; The first sending unit is used to send the target data indicated by the data retransmission request to the external device at the data retransmission time point indicated by the target data retransmission delay; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by using the reward values ​​corresponding to several data retransmission actions under several network packet loss states as rewards through reinforcement learning.

10. A data delayed retransmission device, characterized in that: Applicable to a second electronic device, comprising: A second acquisition unit, configured to acquire a current network packet loss state in response to a data packet loss condition; A second sending unit is used to carry the current network packet loss status in a data retransmission request and send it to an external device, wherein the current network packet loss status is used for the external device to determine a target data retransmission delay allocated to the current network packet loss status according to a target correspondence between a data retransmission action and the network packet loss status, and to send the target data indicated by the data retransmission request at a data retransmission time point indicated by the target data retransmission delay; A receiving unit is used to receive the target data retransmitted by the external device; wherein the target correspondence between the data retransmission action and the network packet loss state is obtained by the external device performing reinforcement learning by using the reward values ​​corresponding to several data retransmission actions respectively under several network packet loss states as rewards.

11. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to perform the operations in the data delayed retransmission method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the data delayed retransmission method according to any one of claims 1 to 8.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the data delayed retransmission method according to any one of claims 1 to 8 are implemented.

Images