Data packet loss compensation and packet loss compensation neural network model training method and device

Abstract

The invention provides a data packet loss compensation method and device and a packet loss compensation neural network model training method and device, and the data packet loss compensation method comprises the steps: receiving a current data packet, obtaining a pre-stored historical data packet under the condition of determining that the current data packet is subjected to packet loss or receiving errors, and storing the historical data packet in the current data packet; the historical data packets are N continuous data packets before the current data packet, and N is a positive integer; inputting the historical data packet into a preset packet loss compensation neural network model to obtain prediction data output by the packet loss compensation neural network model; and performing packet loss compensation processing on the current data packet by adopting the prediction data. Therefore, the lost data packet or the data packet which is wrongly received can be predicted based on the packet loss compensation neural network model, so that the lost data packet is complemented, the integrity of data packet transmission is ensured, and the reliability of data transmission is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning technology, and more particularly to a method and apparatus for packet loss compensation and training a packet loss compensation neural network model. Background Art

[0002] Packet loss compensation is an important signal processing process that aims to recover signals that are lost or damaged due to various reasons, thereby ensuring signal integrity and availability. This technology is widely used in a variety of signal transmission devices and communication scenarios, including but not limited to Bluetooth, Wi-Fi, satellite communications, etc.

[0003] At present, most packet loss compensation technologies rely on simple strategies, such as inserting all-zero signals or copying the previous frame signal to fill the lost part. However, these packet loss compensation technologies based on traditional methods often have unsatisfactory effects or large distortion, and cannot effectively meet the needs of high-quality signal transmission. With the continuous advancement of technology and the increasing market requirements for signal quality, these traditional methods have gradually become incapable of adapting to the complexity and diversity of modern communication systems. Summary of the invention

[0004] The embodiments of the present application provide a method and device for packet loss compensation and a training method for a packet loss compensation neural network model to solve the technical problem that packet loss compensation technology based on traditional methods in related technologies often has unsatisfactory effects or large distortion and cannot effectively meet the needs of high-quality signal transmission.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for compensating for packet loss, the method comprising:

[0007] Receive a current data packet, and if it is determined that the current data packet is lost or received incorrectly, obtain a pre-stored historical data packet, where the historical data packet is N consecutive data packets before the current data packet, where N is a positive integer;

[0008] Inputting the historical data packets into a preset packet loss compensation neural network model to obtain prediction data output by the packet loss compensation neural network model;

[0009] The predicted data is used to perform packet loss compensation processing on the current data packet.

[0010] Optionally, the packet loss compensation neural network model includes: a convolutional network and a recurrent neural network;

[0011] The convolutional network includes multiple convolutional layers, wherein at least some of the convolutional layers are depth-wise convolutional layers, and the convolutional network is used to extract a local feature sequence corresponding to the historical data packet;

[0012] The recurrent neural network is used to summarize the overall trend of the historical data packets based on the local feature sequence, and to perform screening based on the local feature sequence to obtain screened local features, and to obtain the predicted data based on the overall trend and the screened local features.

[0013] Optionally, the multi-layer convolutional layer is composed of a first convolutional layer, a plurality of second convolutional layers and a plurality of third convolutional layers connected in sequence, wherein:

[0014] The number of convolution kernels of the first convolution layer is different from the number of convolution kernels of the second convolution layer and the third convolution layer; and / or,

[0015] The number of channels of the third convolutional layer is different from the number of channels of the first convolutional layer and the second convolutional layer; and / or,

[0016] The stride number of some convolutional layers within the multi-layer convolutional layers is different from the stride number of other convolutional layers; and / or,

[0017] One or more skip connections are used in the multi-layer convolutional layers.

[0018] Optionally, the second convolution layer and the third convolution layer are Depth-wise convolution layers.

[0019] Optionally, the number of channels of the first convolutional layer is 8, the number of convolution kernels is 7, and the number of strides is 2;

[0020] The second convolutional layer has 8 channels, 3 convolution kernels, and 1 stride.

[0021] The number of channels in the third convolutional layer is 16, the number of convolution kernels is 3, and the stride number of the first third convolutional layer in the multiple third convolutional layers is 2, and the stride number of the remaining third convolutional layers is 1.

[0022] Optionally, the multi-layer convolutional layer has a total of 11 layers, wherein a single jump connection is used between the input end of the second convolutional layer and the output end of the third convolutional layer, the input end of the fourth convolutional layer and the output end of the fifth convolutional layer, the input end of the eighth convolutional layer and the output end of the ninth convolutional layer, and the input end of the tenth convolutional layer and the output end of the eleventh convolutional layer;

[0023] The recurrent neural network includes a Dense layer and a GRU recurrent neural network layer, wherein the GRU recurrent neural network layer has 2 layers, and the Dense layer outputs the predicted data.

[0024] Optionally, the prediction data output by the packet loss compensation neural network model includes first prediction data, second prediction data and third prediction data;

[0025] The first prediction data is prediction data corresponding to the current data packet;

[0026] The second prediction data is prediction data of a predetermined length corresponding to the current data packet;

[0027] The third prediction data is prediction data of a predetermined length corresponding to the current data packet;

[0028] Using the predicted data to perform packet loss compensation processing on the current data packet includes:

[0029] Using the first predicted data as a current data packet;

[0030] Based on the second predicted data and a previous data packet adjacent to the current data packet, smoothing the current data packet and the previous data packet;

[0031] Based on the third prediction data and a subsequent data packet adjacent to the current data packet, smoothing processing is performed on the current data packet and the subsequent data packet.

[0032] Optionally, also include:

[0033] The predicted data is used to perform packet loss compensation processing on the current data packet, and then the historical data packet is updated, wherein:

[0034] The current data packet obtained by the packet loss compensation process and the continuous N-1 data packets before the current data packet are stored as the historical data packets.

[0035] In a second aspect, an embodiment of the present application provides a method for training a packet loss compensation neural network model, the method comprising:

[0036] Acquire multiple groups of sample data, wherein each group of the sample data includes: an actual data packet, and the first N consecutive data packets of the actual data packet, where N is a positive integer;

[0037] The first N consecutive data packets of the actual data packet are input into the packet loss compensation neural network model to be trained to obtain the predicted data corresponding to the actual data packet output by the packet loss compensation neural network model to be trained, and the parameters of the packet loss compensation neural network model to be trained are optimized according to the predicted data and the loss function of the actual data packet.

[0038] Optionally, the loss function of the packet loss compensation neural network model is:

[0039] Among them, the y i is the predicted data, the g i is the actual data packet corresponding to the predicted data, and the MSE is the loss value of the packet loss compensation neural network model in the training stage.

[0040] Optionally, the packet loss compensation neural network model includes: a convolutional network and a recurrent neural network;

[0041] The convolutional network includes multiple convolutional layers, wherein at least some of the convolutional layers are depth-wise convolutional layers, and the convolutional network is used to extract the local feature sequence corresponding to the historical data packet;

[0042] The recurrent neural network is used to summarize the overall trend of the historical data packets based on the local feature sequence, and to perform screening based on the local feature sequence to obtain screened local features, and to obtain the predicted data based on the overall trend and the screened local features.

[0043] Optionally, the multi-layer convolutional layer is composed of a first convolutional layer, a plurality of second convolutional layers and a plurality of third convolutional layers connected in sequence, wherein:

[0044] The number of convolution kernels of the first convolution layer is different from the number of convolution kernels of the second convolution layer and the third convolution layer; and / or,

[0045] The number of channels of the third convolutional layer is different from the number of channels of the first convolutional layer and the second convolutional layer; and / or,

[0046] The stride number of some convolutional layers within the multi-layer convolutional layers is different from the stride number of other convolutional layers; and / or,

[0047] One or more skip connections are used in the multi-layer convolutional layers.

[0048] Optionally, the second convolution layer and the third convolution layer are Depth-wise convolution layers.

[0049] Optionally, the number of channels of the first convolutional layer is 8, the number of convolution kernels is 7, and the number of strides is 2;

[0050] The second convolutional layer has 8 channels, 3 convolution kernels, and 1 stride.

[0051] The number of channels in the third convolutional layer is 16, the number of convolution kernels is 3, and the stride number of the first third convolutional layer in the multiple third convolutional layers is 2, and the stride number of the remaining third convolutional layers is 1.

[0052] Optionally, the multi-layer convolutional layer has a total of 11 layers, wherein a single jump connection is used between the input end of the second convolutional layer and the output end of the third convolutional layer, the input end of the fourth convolutional layer and the output end of the fifth convolutional layer, the input end of the eighth convolutional layer and the output end of the ninth convolutional layer, and the input end of the tenth convolutional layer and the output end of the eleventh convolutional layer;

[0053] The recurrent neural network includes a Dense layer and a GRU recurrent neural network layer, wherein the GRU recurrent neural network layer has 2 layers, and the Dense layer outputs the predicted data.

[0054] In a third aspect, an embodiment of the present application provides a device for compensating for data packet loss, the device comprising:

[0055] A receiving module, used to receive a current data packet, and when it is determined that the current data packet is lost or received incorrectly, obtain a pre-stored historical data packet, wherein the historical data packet is N consecutive data packets before the current data packet, where N is a positive integer;

[0056] A first execution module, used for inputting the historical data packet into a preset packet loss compensation neural network model to obtain prediction data output by the packet loss compensation neural network model;

[0057] The first execution module is further configured to use the predicted data to perform packet loss compensation processing on the current data packet.

[0058] In a fourth aspect, an embodiment of the present application provides a training device for a packet loss compensation neural network model, characterized in that the device comprises:

[0059] An acquisition module, used for acquiring multiple groups of sample data, wherein each group of the sample data includes: an actual data packet, and the first N consecutive data packets of the actual data packet, where N is a positive integer;

[0060] The second execution module is used to input the first N consecutive data packets of the actual data packet into the packet loss compensation neural network model to be trained, obtain the predicted data corresponding to the actual data packet output by the packet loss compensation neural network model to be trained, and optimize the parameters of the packet loss compensation neural network model to be trained according to the predicted data and the loss function of the actual data packet.

[0061] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method for compensating for data packet loss as described in the first aspect, or, when executed by the processor, implements the steps of the method for training a neural network model for packet loss compensation as described in the second aspect.

[0062] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for compensating for data packet loss as described in the first aspect are implemented, or when the computer program is executed by the processor, the steps of the method for training a neural network model for packet loss compensation as described in the second aspect are implemented.

[0063] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method for compensating for data packet loss as described in the first aspect, or, when executed by the processor, implement the steps of the method for training a neural network model for packet loss compensation as described in the second aspect.

[0064] In the embodiment of the present application, by using the information of historical data packets and combining the packet loss compensation neural network model for prediction, it is possible to effectively fill in the lost data packets or receive erroneous data packets, thereby ensuring the integrity of data transmission. This method is more accurate than the prior art method of filling in the lost part by inserting all-zero signals or copying the previous frame signal, and can ensure the integrity of data packet transmission, improve the reliability of data transmission, and significantly reduce the delay of data transmission, especially in real-time application scenarios (such as video streaming, online games, etc.), which can provide a smoother user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0066] Figure 1 A flowchart of a method for compensating for data packet loss provided in an embodiment of the present application;

[0067] Figure 2 A schematic diagram of the structure of a packet loss compensation neural network model provided in an embodiment of the present application;

[0068] Figure 3A flowchart of a method for training a packet loss compensation neural network model provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of a training method for a packet loss compensation neural network model provided in an embodiment of the present application;

[0070] Figure 5 A structural block diagram of a device for compensating for data packet loss provided in an embodiment of the present application;

[0071] Figure 6 A structural block diagram of a training device for a packet loss compensation neural network model provided in an embodiment of the present application;

[0072] Figure 7 A structural block diagram of a network device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] Figure 1 A method for compensating for packet loss according to an embodiment of the present application is shown. Figure 1 As shown, the method includes:

[0075] Step S101, receiving a current data packet, and obtaining a pre-stored historical data packet when it is determined that the current data packet is lost or received incorrectly;

[0076] The historical data packet is the N consecutive data packets before the current data packet, where N is a positive integer;

[0077] Step S102: inputting historical data packets into a preset packet loss compensation neural network model to obtain prediction data output by the packet loss compensation neural network model;

[0078] Step S103: Use the predicted data to perform packet loss compensation processing on the current data packet.

[0079] In step S101, the receiving end receives a data packet and performs preliminary processing on the received data packet. For example, when it is determined that a data packet that should be received is not received, or a data packet is received, but a data reception error is confirmed by parsing the data packet header and the payload, that is, when it is determined that a packet loss or reception error occurs, the receiving end will enter a compensation process to perform packet loss compensation processing on the current data packet. When it is confirmed that a packet loss or reception error occurs in the current data packet, the receiving end will extract N consecutive data packets (N is a positive integer) that are pre-stored from the memory. The N consecutive data packets refer to the data packets before the current data packet. For example, in the data stream, if the current data packet is the i-th data packet, the N consecutive data packets are the i-1th data packet, the i-2th data packet, and the iNth data packet.

[0080] It is understandable that the data packet may be an audio data packet, a video data packet or other types of data packets. The value of N may be configured as needed according to the actual application scenario. This application does not specifically limit this. In addition, the embodiments of the present invention are also applicable to various application scenarios of wired or wireless signal transmission.

[0081] In step S102, historical data packets are taken as input data and sent to a preset packet loss compensation neural network model. The packet loss compensation neural network model can use its learned patterns and features to analyze and extract features of historical data packets to generate prediction data for current lost or erroneous data packets. This process improves the accuracy of packet loss compensation and makes the prediction data more reliable.

[0082] In step S103, the predicted data output by the packet loss compensation neural network model can be used to replace the lost or erroneous current data packet, and the receiving end can use the predicted data as a valid data packet for subsequent data processing, thereby ensuring the continuity of the data flow.

[0083] In summary, the method for data packet loss compensation provided in the embodiment of the present application can obtain pre-stored historical data packets during the wireless signal transmission process, when it is determined that the current data packet has been lost or received incorrectly, and input them into the packet loss compensation neural network model for prediction, which can effectively fill the lost data packets or the received erroneous data packets, thereby ensuring the integrity of data transmission. This method is more accurate than the prior art method of filling the lost part by inserting an all-zero signal or copying the previous frame signal, can ensure the integrity of data packet transmission, improve the reliability of data transmission, and can significantly reduce the delay of data transmission, especially in real-time application scenarios (such as video streaming, online games, etc.), can provide a smoother user experience.

[0084] It should be noted that, in a possible implementation, the packet loss compensation neural network model includes: a convolutional network and a recurrent neural network; the convolutional network includes multiple convolutional layers, at least some of which are depth-wise convolutional layers, and the convolutional network is used to extract local feature sequences corresponding to historical data packets; the recurrent neural network is used to summarize the overall trend of historical data packets based on the local feature sequence, and to filter based on the local feature sequence to obtain the filtered local features, and to obtain predicted data based on the overall trend and the filtered local features.

[0085] At present, depth-wise convolutional networks have been verified to have obvious advantages over traditional convolutional networks in reducing the number of parameters and computational complexity. They can reduce the number of parameters and computing resource consumption of neural network models to achieve the goal of lightweight.

[0086] The structure of the preset packet loss compensation neural network model provided in the embodiment of the present application is now described, and reference may be made to Figure 2 .

[0087] In a possible implementation, the multi-layer convolutional layer is composed of a first convolutional layer, multiple second convolutional layers and multiple third convolutional layers connected in sequence, wherein the number of convolution kernels of the first convolutional layer is different from the number of convolution kernels of the second convolutional layer and the third convolutional layer, respectively; and / or the number of channels of the third convolutional layer is different from the number of channels of the first convolutional layer and the second convolutional layer, respectively; and / or the stride number of some convolutional layers in the multi-layer convolutional layer is different from the stride number of other convolutional layers; and / or one or more skip connections are used in the multi-layer convolutional layer.

[0088] In a possible implementation, the second convolution layer and the third convolution layer are depth-wise convolution layers. Therefore, this embodiment can play a good role in amplifying and extracting the local features of the signal through feature extraction work of a multi-layer depth-wise convolution network.

[0089] For convolutional networks, the number of convolution kernels, the number of channels, and the number of strides will affect its performance in terms of computational complexity, number of parameters, and detailed feature extraction. Skip connections help alleviate the gradient vanishing problem in deep networks, allowing information to flow through different paths in the network, thereby improving the training efficiency and performance of the model. They can also help retain low-level features and enhance the expressiveness of the model.

[0090] In the design of the multi-layer convolutional layers of this embodiment, by respectively configuring the number of convolution kernels, the number of channels, the stride and introducing jump connections of the first convolutional layer, the second convolutional layer and the third convolutional layer, the original one-dimensional signal data can be changed into a two-dimensional signal with richer detail features per unit time through multiple calculations. At the same time, through the multi-layer network, the local detail features in the signal are gradually amplified.

[0091] In a possible implementation, the number of channels of the first convolution layer is 8, the number of convolution kernels is 7, and the stride is 2; the number of channels of the second convolution layer is 8, the number of convolution kernels is 3, and the stride is 1; the number of channels in the third convolution layer is 16, the number of convolution kernels is 3, and the stride of the first third convolution layer in the plurality of third convolution layers is 2, and the stride of the remaining third convolution layers is 1. The network layer with a stride of 2 can reduce the sequence length of the input signal and reduce the computational complexity of the subsequent network.

[0092] It should be noted that the above-mentioned design of the number of channels, number of convolution kernels, and number of strides is only an exemplary description and can be adjusted according to actual needs.

[0093] In a possible implementation, the multi-layer convolutional layer has a total of 11 layers, wherein a jump connection is used between the input end of the second convolutional layer and the output end of the third convolutional layer, between the input end of the fourth convolutional layer and the output end of the fifth convolutional layer, between the input end of the eighth convolutional layer and the output end of the ninth convolutional layer, and between the input end of the tenth convolutional layer and the output end of the eleventh convolutional layer.

[0094] In a possible implementation, the recurrent neural network includes a Dense layer (fully connected layer) and a GRU (Gate Recurrent Unit) recurrent neural network layer, wherein the GRU recurrent neural network layer has two layers, and the Dense layer outputs the predicted data.

[0095] In a preferred embodiment, it can be designed as follows Figure 2 A packet loss compensation neural network model including 11 layers of convolutional networks and 2 layers of GRU recurrent neural networks. Among them, in the 11-layer convolutional network, the first layer is a normal convolutional network layer, and the remaining 10 layers are depth-wise convolutional network layers. In addition, the first and sixth layers are designed with a stride of 2, and a single jump connection is used in the second, third, fourth, fifth, eighth, ninth, and tenth and eleventh layers to extract more detailed features. After the feature extraction work of the multi-layer depth-wise convolutional network, the local features of the signal can be well amplified and extracted. And because the weights of each channel kernel of the convolutional network are shared, it operates through a sliding window in global calculation, which can effectively control the expansion of the parameter quantity.

[0096] The GRU recurrent neural network is used to summarize and filter the front and back features of the sequence data through the recurrent neural network after extracting the local features of the first N consecutive data packets by the multi-layer Depth-wise convolutional network. By summarizing the overall trend of the data of the first N consecutive data packets, the overall structure of the target data packet (the data packet corresponding to the lost data packet or the transmission error data packet) is predicted, and by filtering the local features of the first N consecutive data packets, the details of the target data packet are depicted, thereby reducing the difference between the actual lost data packet or the transmission error data packet as a whole. The characteristic of the GRU recurrent network is that it can extract the global features of the sequence signal. By introducing the gating mechanism, it can effectively capture the relevant information in the past sequence when calculating the current information on the sequence, and effectively retain and discard the influence of the past information in the sequence on the information at the current moment, so as to extract the front and back information in a sequence more holistically, especially the dependency relationship of the front and back information with a larger span than the convolutional network.

[0097] Figure 2 A Dense Layer (fully connected layer) is used at the end of the packet loss compensation neural network model shown, which is used to fully connect the input data and the output results in the neural network and output the predicted data.

[0098] The packet loss compensation neural network model provided in the embodiment of the present invention can be deployed on low-power edge devices due to its small number of parameters and low consumption of computing resources. For example, it can be applied to various headphones, speakers, watches, bracelets, glasses, VR devices and many other devices.

[0099] In a possible implementation, the prediction data output by the packet loss compensation neural network model includes first prediction data, second prediction data and third prediction data; the first prediction data is prediction data corresponding to the current data packet; the second prediction data is prediction data corresponding to a predetermined length before the current data packet; the third prediction data is prediction data corresponding to a predetermined length after the current data packet;

[0100] Correspondingly, step S103 uses the predicted data to perform packet loss compensation processing on the current data packet, including: taking the first predicted data as the current data packet; based on the second predicted data and the previous data packet adjacent to the current data packet, smoothing the current data packet and the previous data packet; based on the third predicted data and the next data packet adjacent to the current data packet, smoothing the current data packet and the next data packet.

[0101] It should be noted that the predetermined length may be 1 / 2, 1 / 3, 1 / 4, etc., of the length of the data in the current data packet, and may be configured according to actual needs.

[0102] When compensating the current data packet, the second predicted data and the previous data packet adjacent to the current data packet can be combined for smoothing. Through smoothing, the mutations between data packets can be reduced, ensuring the smoothness and consistency of the data stream, helping to reduce the distortion caused by packet loss compensation, and making the data more coherent. Similarly, smoothing can be performed based on the third predicted data and the next data packet adjacent to the current data packet. This process ensures that the transition between the current data packet and the next data packet is also smooth, further enhancing the continuity and stability of the data stream. In this way, the packet loss compensation neural network model can better adapt to changes in the data stream and reduce fluctuations caused by packet loss compensation.

[0103] In specific application scenarios, experimental results show that the first predicted data output by the packet loss compensation neural network model is significantly closer to the actual lost data packet, and by using multiple predicted data of preset length (such as 1 / 4 length) for smoothing, the connection between the lost data packet and its previous and subsequent data can be more in line with the actual situation than the traditional method, and is more difficult to detect when using audio as the test signal, thus achieving the expected effect.

[0104] In a possible implementation, the method further includes: using the predicted data to perform packet loss compensation processing on the current data packet, and then updating the historical data packet, wherein the current data packet obtained by the packet loss compensation processing and the N-1 consecutive data packets before the current data packet are stored as historical data packets. Thus, by updating the historical data packets, the receiving end can retain the latest, compensated data, which helps to maintain the integrity of the data in subsequent processing and analysis, and the updated historical data packets can serve as the basis for future predictions, ensuring that the packet loss compensation neural network model can refer to the latest state when making decisions, and when encountering continuous packet loss, the prediction of the next lost data packet can be continued through the last predicted lost data packet, solving the problem of continuous packet loss.

[0105] It is understandable that, when the current data packet is correctly received, the correctly received current data packet can be used to update the historical data packet, so that the N data packets in the historical data packet are continuously updated as new data packets are received.

[0106] Figure 3 A training method for a packet loss compensation neural network model provided in accordance with an embodiment of the present application is shown, such as Figure 3 As shown, the method includes:

[0107] Step S301, obtaining multiple groups of sample data;

[0108] Each set of sample data includes: an actual data packet, and the first N consecutive data packets of the actual data packet, where N is a positive integer;

[0109] Step S302: input the first N consecutive data packets of the actual data packet into the packet loss compensation neural network model to be trained, obtain the predicted data corresponding to the actual data packet output by the packet loss compensation neural network model to be trained, and optimize the parameters of the packet loss compensation neural network model to be trained according to the loss function of the predicted data and the actual data packet.

[0110] Among them, the loss function of the packet loss compensation neural network model is:

[0111] Among them, y i is the predicted data, g i is the actual data packet corresponding to the predicted data, and MSE is the loss value of the packet loss compensation neural network model in the training stage.

[0112] It should be noted that the training method of the packet loss compensation neural network model is as follows Figure 4 As shown, by obtaining multiple groups of sample data and inputting the first N consecutive data packets of the actual data packets into the packet loss compensation neural network model to be trained, the model can learn the relationship between the data packets and generate corresponding prediction data. By calculating the loss function and optimizing the model parameters, it can ultimately achieve effective compensation for packet loss. This process not only improves the prediction accuracy of the model, but also enhances its robustness in practical applications.

[0113] In one possible implementation, the packet loss compensation neural network model includes: a convolutional network and a recurrent neural network; the convolutional network includes multiple convolutional layers, at least some of which are depth-wise convolutional layers, and the convolutional network is used to extract local feature sequences corresponding to historical data packets; the recurrent neural network is used to summarize the overall trend of historical data packets based on the local feature sequences, and to filter based on the local feature sequences to obtain filtered local features, and to obtain predicted data based on the overall trend and the filtered local features.

[0114] At present, depth-wise convolutional networks have been verified to have obvious advantages over traditional convolutional networks in reducing the number of parameters and computational complexity. They can reduce the number of parameters and computing resource consumption of neural network models to achieve the goal of lightweight.

[0115] The structure of the preset packet loss compensation neural network model provided in the embodiment of the present application is now described, and reference may be made to Figure 2 .

[0116] In a possible implementation, the multi-layer convolutional layer is composed of a first convolutional layer, multiple second convolutional layers and multiple third convolutional layers connected in sequence, wherein the number of convolution kernels of the first convolutional layer is different from the number of convolution kernels of the second convolutional layer and the third convolutional layer, respectively; and / or the number of channels of the third convolutional layer is different from the number of channels of the first convolutional layer and the second convolutional layer, respectively; and / or the number of strides of some convolutional layers in the multi-layer convolutional layer is different from the number of strides of other convolutional layers; and / or one or more jump connections are used in the multi-layer convolutional layer.

[0117] In a possible implementation, the second convolution layer and the third convolution layer are depth-wise convolution layers. Therefore, this embodiment can play a good role in amplifying and extracting the local features of the signal through feature extraction work of a multi-layer depth-wise convolution network.

[0118] For convolutional networks, the number of convolution kernels, the number of channels, and the number of strides will affect its performance in terms of computational complexity, number of parameters, and detailed feature extraction. Skip connections help alleviate the gradient vanishing problem in deep networks, allowing information to flow through different paths in the network, thereby improving the training efficiency and performance of the model. They can also help retain low-level features and enhance the expressiveness of the model.

[0119] In the design of the multi-layer convolutional layers of this embodiment, by respectively configuring the number of convolution kernels, the number of channels, the stride and introducing jump connections of the first convolutional layer, the second convolutional layer and the third convolutional layer, the original one-dimensional signal data can be changed into a two-dimensional signal with richer detail features per unit time through multiple calculations. At the same time, through the multi-layer network, the local detail features in the signal are gradually amplified.

[0120] In a possible implementation, the number of channels of the first convolution layer is 8, the number of convolution kernels is 7, and the stride is 2; the number of channels of the second convolution layer is 8, the number of convolution kernels is 3, and the stride is 1; the number of channels in the third convolution layer is 16, the number of convolution kernels is 3, and the stride of the first third convolution layer in the plurality of third convolution layers is 2, and the stride of the remaining third convolution layers is 1. The network layer with a stride of 2 can reduce the sequence length of the input signal and reduce the computational complexity of the subsequent network.

[0121] It should be noted that the above-mentioned design of the number of channels, number of convolution kernels, and number of strides is only an exemplary description and can be adjusted according to actual needs.

[0122] In a possible implementation, the multi-layer convolutional layer has a total of 11 layers, wherein a jump connection is used between the input end of the second convolutional layer and the output end of the third convolutional layer, between the input end of the fourth convolutional layer and the output end of the fifth convolutional layer, between the input end of the eighth convolutional layer and the output end of the ninth convolutional layer, and between the input end of the tenth convolutional layer and the output end of the eleventh convolutional layer.

[0123] In a possible implementation, the recurrent neural network includes a Dense layer and a GRU recurrent neural network layer, wherein the GRU recurrent neural network layer has two layers, and the Dense layer outputs the predicted data.

[0124] In a preferred embodiment, it can be designed as follows Figure 2 A packet loss compensation neural network model including 11 layers of convolutional networks and 2 layers of GRU recurrent neural networks. Among them, in the 11-layer convolutional network, the first layer is a normal convolutional network layer, and the remaining 10 layers are depth-wise convolutional network layers. In addition, the first and sixth layers are designed with a stride of 2, and a single jump connection is used in the second, third, fourth, fifth, eighth, ninth, and tenth and eleventh layers to extract more detailed features. After the feature extraction work of the multi-layer depth-wise convolutional network, the local features of the signal can be well amplified and extracted. And because the weights of each channel kernel of the convolutional network are shared, it operates through a sliding window in global calculation, which can effectively control the expansion of the parameter quantity.

[0125] The GRU recurrent neural network is used to summarize and filter the front and back features of the sequence data through the recurrent neural network after extracting the local features of the first N consecutive data packets by the multi-layer Depth-wise convolutional network. By summarizing the overall trend of the data of the first N consecutive data packets, the overall structure of the target data packet (the data packet corresponding to the lost data packet or the transmission error data packet) is predicted, and by filtering the local features of the first N consecutive data packets, the details of the target data packet are depicted, thereby reducing the difference between the actual lost data packet or the transmission error data packet as a whole. The characteristic of the GRU recurrent network is that it can extract the global features of the sequence signal. By introducing the gating mechanism, it can effectively capture the relevant information in the past sequence when calculating the current information on the sequence, and effectively retain and discard the influence of the past information in the sequence on the information at the current moment, so as to extract the front and back information in a sequence more holistically, especially the dependency relationship of the front and back information with a larger span than the convolutional network.

[0126] Figure 2A Dense Layer (fully connected layer) is used at the end of the packet loss compensation neural network model shown, which is used to fully connect the input data and the output results in the neural network and output the predicted data.

[0127] The packet loss compensation neural network model provided in the embodiment of the present invention can be deployed on low-power edge devices due to its small number of parameters and low consumption of computing resources. For example, it can be applied to various headphones, speakers, watches, bracelets, glasses, VR devices and many other devices.

[0128] Figure 5 A device for compensating for packet loss according to an embodiment of the present application is shown. Figure 5 As shown, the device 50 includes:

[0129] The receiving module 501 is used to receive the current data packet, and when it is determined that the current data packet is lost or received incorrectly, obtain a pre-stored historical data packet, where the historical data packet is N consecutive data packets before the current data packet, where N is a positive integer;

[0130] The first execution module 502 is used to input the historical data packets into a preset packet loss compensation neural network model to obtain the prediction data output by the packet loss compensation neural network model;

[0131] The first execution module 502 is further configured to use the predicted data to perform packet loss compensation processing on the current data packet.

[0132] In summary, by using the information of historical data packets and combining it with the packet loss compensation neural network model for prediction, it is possible to effectively fill in lost data packets or receive erroneous data packets, thereby ensuring the integrity of data transmission. This method is more accurate than the existing technology of filling in the lost part by inserting all-zero signals or copying the previous frame signal. It can ensure the integrity of data packet transmission, improve the reliability of data transmission, and significantly reduce the delay of data transmission, especially in real-time application scenarios (such as video streaming, online games, etc.), which can provide a smoother user experience.

[0133] Figure 6 A training device for a packet loss compensation neural network model provided in accordance with an embodiment of the present application is shown. Figure 6 As shown, the device 60 includes:

[0134] The acquisition module 601 is used to acquire multiple groups of sample data, wherein each group of sample data includes: an actual data packet, and the first N consecutive data packets of the actual data packet, where N is a positive integer;

[0135] The second execution module 602 is used to input the first N consecutive data packets of the actual data packet into the packet loss compensation neural network model to be trained, obtain the predicted data corresponding to the actual data packet output by the packet loss compensation neural network model to be trained, and optimize the parameters of the packet loss compensation neural network model to be trained according to the loss function of the predicted data and the actual data packet.

[0136] In summary, by using the information of historical data packets and combining it with the packet loss compensation neural network model for prediction, it is possible to effectively fill in lost data packets or receive erroneous data packets, thereby ensuring the integrity of data transmission. This method is more accurate than the existing technology of filling in the lost part by inserting all-zero signals or copying the previous frame signal. It can ensure the integrity of data packet transmission, improve the reliability of data transmission, and significantly reduce the delay of data transmission, especially in real-time application scenarios (such as video streaming, online games, etc.), which can provide a smoother user experience.

[0137] The present application embodiment provides an electronic device 70, such as Figure 7 As shown, the electronic device 70 includes: a processor 701, a memory 702, and a program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, the steps of the packet loss compensation and packet loss compensation neural network model training method shown in the above embodiment are implemented.

[0138] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the packet loss compensation and packet loss compensation neural network model training method shown in the above embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0139] The embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the data packet loss compensation and packet loss compensation neural network model training method shown in the above method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0140] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0142] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for compensating for packet loss, characterized in that: The method comprises: Receive a current data packet, and if it is determined that the current data packet is lost or received incorrectly, obtain a pre-stored historical data packet, where the historical data packet is N consecutive data packets before the current data packet, where N is a positive integer; Inputting the historical data packets into a preset packet loss compensation neural network model to obtain prediction data output by the packet loss compensation neural network model; The predicted data is used to perform packet loss compensation processing on the current data packet.

2. The method according to claim 1, characterized in that The packet loss compensation neural network model includes: a convolutional network and a recurrent neural network; The convolutional network includes multiple convolutional layers, wherein at least some of the convolutional layers are depth-wise convolutional layers, and the convolutional network is used to extract a local feature sequence corresponding to the historical data packet; The recurrent neural network is used to summarize the overall trend of the historical data packets based on the local feature sequence, and to perform screening based on the local feature sequence to obtain screened local features, and to obtain the predicted data based on the overall trend and the screened local features.

3. The method according to claim 2, characterized in that The multi-layer convolutional layer is composed of a first convolutional layer, a plurality of second convolutional layers and a plurality of third convolutional layers connected in sequence, wherein: The number of convolution kernels of the first convolution layer is different from the number of convolution kernels of the second convolution layer and the third convolution layer; and / or, The number of channels of the third convolutional layer is different from the number of channels of the first convolutional layer and the second convolutional layer; and / or, The stride number of some convolutional layers within the multi-layer convolutional layers is different from the stride number of other convolutional layers; and / or, One or more skip connections are used in the multi-layer convolutional layers.

4. The method according to claim 3, characterized in that The second convolution layer and the third convolution layer are Depth-wise convolution layers.

5. The method according to claim 4, characterized in that The number of channels of the first convolutional layer is 8, the number of convolution kernels is 7, and the number of strides is 2; The second convolutional layer has 8 channels, 3 convolution kernels, and 1 stride. The number of channels in the third convolutional layer is 16, the number of convolution kernels is 3, and the stride number of the first third convolutional layer in the multiple third convolutional layers is 2, and the stride number of the remaining third convolutional layers is 1.

6. The method according to claim 4, characterized in that The multi-layer convolutional layer has a total of 11 layers, wherein a single jump connection is used between the input end of the second convolutional layer and the output end of the third convolutional layer, the input end of the fourth convolutional layer and the output end of the fifth convolutional layer, the input end of the eighth convolutional layer and the output end of the ninth convolutional layer, and the input end of the tenth convolutional layer and the output end of the eleventh convolutional layer; The recurrent neural network includes a Dense layer and a GRU recurrent neural network layer, wherein the GRU recurrent neural network layer has 2 layers, and the Dense layer outputs the predicted data.

7. The method according to claim 1, characterized in that The prediction data output by the packet loss compensation neural network model includes first prediction data, second prediction data and third prediction data; The first prediction data is prediction data corresponding to the current data packet; The second prediction data is prediction data of a predetermined length corresponding to the current data packet; The third prediction data is prediction data of a predetermined length corresponding to the current data packet; Using the predicted data to perform packet loss compensation processing on the current data packet includes: Using the first predicted data as a current data packet; Based on the second predicted data and a previous data packet adjacent to the current data packet, smoothing the current data packet and the previous data packet; Based on the third prediction data and a subsequent data packet adjacent to the current data packet, smoothing processing is performed on the current data packet and the subsequent data packet.

8. The method according to any one of claims 1 to 7, characterized in that Also includes: The predicted data is used to perform packet loss compensation processing on the current data packet, and then the historical data packet is updated, wherein: The current data packet obtained by the packet loss compensation process and the continuous N-1 data packets before the current data packet are stored as the historical data packets.

9. A method for training a packet loss compensation neural network model, characterized in that: The method comprises: Acquire multiple groups of sample data, wherein each group of the sample data includes: an actual data packet, and the first N consecutive data packets of the actual data packet, where N is a positive integer; The first N consecutive data packets of the actual data packet are input into the packet loss compensation neural network model to be trained to obtain the predicted data corresponding to the actual data packet output by the packet loss compensation neural network model to be trained, and the parameters of the packet loss compensation neural network model to be trained are optimized according to the predicted data and the loss function of the actual data packet.

10. The method according to claim 9, characterized in that The loss function of the packet loss compensation neural network model is: Among them, the y i is the predicted data, the g i is the actual data packet corresponding to the predicted data, and the MSE is the loss value of the packet loss compensation neural network model in the training stage.

11. The method according to claim 9, characterized in that The packet loss compensation neural network model includes: a convolutional network and a recurrent neural network; The convolutional network includes multiple convolutional layers, wherein at least some of the convolutional layers are depth-wise convolutional layers, and the convolutional network is used to extract the local feature sequence corresponding to the historical data packet; The recurrent neural network is used to summarize the overall trend of the historical data packets based on the local feature sequence, and to perform screening based on the local feature sequence to obtain screened local features, and to obtain the predicted data based on the overall trend and the screened local features.

12. The method according to claim 11, characterized in that The multi-layer convolutional layer is composed of a first convolutional layer, a plurality of second convolutional layers and a plurality of third convolutional layers connected in sequence, wherein: The number of convolution kernels of the first convolution layer is different from the number of convolution kernels of the second convolution layer and the third convolution layer; and / or, The number of channels of the third convolutional layer is different from the number of channels of the first convolutional layer and the second convolutional layer; and / or, The stride number of some convolutional layers within the multi-layer convolutional layers is different from the stride number of other convolutional layers; and / or, One or more skip connections are used in the multi-layer convolutional layers.

13. The method according to claim 12, characterized in that The second convolution layer and the third convolution layer are Depth-wise convolution layers.

14. The method according to claim 13, characterized in that The number of channels of the first convolutional layer is 8, the number of convolution kernels is 7, and the number of strides is 2; The second convolutional layer has 8 channels, 3 convolution kernels, and 1 stride. The number of channels in the third convolutional layer is 16, the number of convolution kernels is 3, and the stride number of the first third convolutional layer in the multiple third convolutional layers is 2, and the stride number of the remaining third convolutional layers is 1.

15. The method according to claim 13, characterized in that The multi-layer convolutional layer has a total of 11 layers, wherein a single jump connection is used between the input end of the second convolutional layer and the output end of the third convolutional layer, the input end of the fourth convolutional layer and the output end of the fifth convolutional layer, the input end of the eighth convolutional layer and the output end of the ninth convolutional layer, and the input end of the tenth convolutional layer and the output end of the eleventh convolutional layer; The recurrent neural network includes a Dense layer and a GRU recurrent neural network layer, wherein the GRU recurrent neural network layer has 2 layers, and the Dense layer outputs the predicted data.

16. A device for compensating for packet loss, characterized in that: The device comprises: A receiving module, used to receive a current data packet, and when it is determined that the current data packet is lost or received incorrectly, obtain a pre-stored historical data packet, wherein the historical data packet is N consecutive data packets before the current data packet, where N is a positive integer; A first execution module, used for inputting the historical data packet into a preset packet loss compensation neural network model to obtain prediction data output by the packet loss compensation neural network model; The first execution module is further configured to use the predicted data to perform packet loss compensation processing on the current data packet.

17. A training device for a packet loss compensation neural network model, characterized in that: The device comprises: An acquisition module, used for acquiring multiple groups of sample data, wherein each group of the sample data includes: an actual data packet, and the first N consecutive data packets of the actual data packet, where N is a positive integer; The second execution module is used to input the first N consecutive data packets of the actual data packet into the packet loss compensation neural network model to be trained, obtain the predicted data corresponding to the actual data packet output by the packet loss compensation neural network model to be trained, and optimize the parameters of the packet loss compensation neural network model to be trained according to the predicted data and the loss function of the actual data packet.

18. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method for compensating for packet loss as described in any one of claims 1 to 8, or the program, when executed by the processor, implements the steps of the method for training a neural network model for packet loss compensation as described in any one of claims 9 to 15.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by the processor, implements the steps of the method for packet loss compensation as described in any one of claims 1 to 8, or, when executed by the processor, implements the steps of the method for training a packet loss compensation neural network model as described in any one of claims 9 to 15.

20. A computer program product, characterized in that It includes computer instructions, which, when executed by the processor, implement the steps of the method for packet loss compensation as described in any one of claims 1-8, or the program, when executed by the processor, implements the steps of the method for training a packet loss compensation neural network model as described in any one of claims 9-15.

Images