Video transmission method and system for router, gateway, IPC and ONU

Abstract

The invention provides a video transmission method and system for a router, a gateway, an IPC and an ONU in the technical field of video transmission, and the method comprises the steps: S1, an edge computing device creates a coding table, a network grading table and a video grading table, and monitors network parameters in real time; s2, acquiring original videos to be transmitted, and performing parallel coding on the original videos based on the coding table, the network grading table and the network parameters to obtain coded videos; s3, matching the transmission priority of each coded video based on the video grading table; s4, dynamically selecting a transmission link based on the network grading table and the network parameters; s5, transmitting the corresponding coded video based on the selected transmission link and the transmission priority; and step S6, in the transmission process of the coded video, the edge computing device performs bandwidth prediction based on the bandwidth prediction model, and dynamically adjusts bandwidth allocation based on a bandwidth prediction result. The method has the advantage that the quality and the stability of video transmission are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video transmission, and particularly to a video transmission method and system for routers, gateways, IP cameras, and ONUs. Background Art

[0002] With the popularization of emerging applications such as 4K / 8K ultra-high-definition videos and AR / VR, network video traffic has shown exponential growth, and users' demand for high-quality video content has also been increasing day by day. This undoubtedly poses new challenges to resource-constrained edge computing devices (such as routers, gateways, IP cameras (network cameras), and ONUs).

[0003] Traditionally, when edge computing devices transmit videos, the following problems exist:

[0004] 1. Network congestion problem: Existing H.264 / AVC encoding is difficult to balance the compression ratio and image quality in a complex network environment, especially prone to mosaic effects when the bandwidth fluctuates.

[0005] 2. Delay sensitivity problem: The video transmission scheme based on the traditional TCP protocol will generate additional delays during packet loss retransmission and cannot meet the QoS requirements of real-time video transmission.

[0006] 3. Low bandwidth utilization: Adopting a static bandwidth allocation strategy is difficult to adapt to a dynamic network environment, resulting in insufficient or overloaded bandwidth utilization. When the bandwidth utilization is overloaded, it will cause video stuttering or buffering. If the video is over-compressed, it will lead to a decline in video picture quality and loss of details, affecting the viewing experience.

[0007] 4. Insufficient multi-network access coordination: The existing network switching technology has too high a delay when switching in a heterogeneous network environment, resulting in video stuttering.

[0008] Therefore, how to provide a video transmission method and system for routers, gateways, IP cameras, and ONUs to improve the quality and stability of video transmission has become an urgent technical problem to be solved. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a video transmission method and system for routers, gateways, IP cameras, and ONUs to improve the quality and stability of video transmission.

[0010] In a first aspect, the present invention provides a video transmission method for routers, gateways, IP cameras, and ONUs, including the following steps:

[0011] Step S1: An edge computing device with a device type of router, gateway, IPC, or ONU creates an encoding table, a network classification table, and a video classification table, and monitors the current network parameters of each transmission link in real time;

[0012] Step S2: The edge computing device obtains the original videos to be transmitted, and performs parallel encoding on each original video based on the encoding table, the network classification table, and the network parameters to obtain the corresponding encoded videos;

[0013] Step S3: The edge computing device matches the transmission priorities of each encoded video based on the video classification table;

[0014] Step S4: The edge computing device dynamically selects a transmission link based on the network classification table and the network parameters;

[0015] Step S5: The edge computing device transmits the corresponding encoded videos based on the selected transmission link and the transmission priority;

[0016] Step S6: During the transmission of the encoded videos, the edge computing device performs bandwidth prediction based on a pre-trained bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

[0017] Further, in Step S1, the encoding table stores the correspondence between the encoding format and the network level; the encoding format at least includes H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP9, VP10, AV1, MPEG-4, AVC-Intra, and ProRes;

[0018] The network classification table stores the correspondence between the network level and the network parameters;

[0019] The video classification table stores the correspondence between the transmission priority and the video content, user account, and playback time limit;

[0020] The transmission link at least includes a wired transmission link, a wireless local area network, a cellular mobile network, Bluetooth, and satellite communication;

[0021] The network parameters at least include network bandwidth, network delay, jitter, packet loss rate, throughput, and TCP connection establishment time.

[0022] Further, Step S2 is specifically as follows:

[0023] The edge computing device obtains the original video to be transmitted, determines the network levels of each parameter link based on the network classification table and network parameters, matches the corresponding encoding format from the encoding table based on the optimal network level, and parallelly encodes each original video through a number of encoders to obtain the corresponding encoded video. During the encoding process, the bit rate is dynamically adjusted by combining the ROI encoding enhancement technology;

[0024] The specific steps of step S3 are as follows:

[0025] The edge computing device obtains the video content, user account, and playback time limit of each encoded video, matches the transmission priority of each encoded video from the video classification table based on the video content, user account, and playback time limit, and sequentially stores each encoded video into the transmission queue based on the transmission priority.

[0026] Further, the specific steps of step S4 are as follows:

[0027] The edge computing device matches the network level corresponding to each transmission link from the network classification table based on the network parameters, dynamically selects the transmission link in the working state based on the network level, and dynamically adjusts the frame rate of the encoded video;

[0028] The specific steps of step S5 are as follows:

[0029] The edge computing device selects the encoded video to be transmitted from the transmission queue based on the transmission priority, and transmits the encoded video to be transmitted through the selected transmission link. 500 ms of the encoded video is pre-cached before transmission.

[0030] Further, the specific steps of step S6 are as follows:

[0031] An ARIMA network and an LSTM network are used to create a bandwidth prediction model. The loss function of the bandwidth prediction model is set as the mean square error function, and a data set is constructed based on a large number of historical network parameters; the ARIMA network is used for short-term bandwidth prediction; the LSTM network is used for long-term bandwidth prediction;

[0032] Divide the dataset into a training set, a validation set, and a test set based on a preset segmentation ratio. Train the bandwidth prediction model using the training set until the loss value of the loss function is less than a preset loss threshold. During the training process, compress the bandwidth prediction model using knowledge distillation technology and dynamic pruning technology. Validate the trained bandwidth prediction model using the validation set to determine whether the prediction accuracy is greater than a preset accuracy threshold. If not, the validation fails, and the training set is expanded for further training. If so, the validation succeeds. Test the bandwidth prediction model that has passed the validation using the test set to determine whether the confidence level is greater than a preset confidence level threshold. If not, the test fails, and the training set is expanded for further training. If so, the test succeeds, training ends, and the bandwidth prediction model is deployed to the edge computing device;

[0033] During the transmission process of the encoded video, the edge computing device performs bandwidth prediction based on the bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

[0034] In a second aspect, the present invention provides a video transmission system for routers, gateways, IP cameras, and ONUs, including the following modules:

[0035] An initialization module for creating an encoding table, a network classification table, and a video classification table for an edge computing device with a device type of router, gateway, IP camera, or ONU, and for real-time monitoring of the current network parameters of each transmission link;

[0036] A parallel encoding module for the edge computing device to obtain the original videos to be transmitted, and perform parallel encoding on each original video based on the encoding table, the network classification table, and the network parameters to obtain corresponding encoded videos;

[0037] A transmission priority matching module for the edge computing device to match the transmission priorities of the encoded videos based on the video classification table;

[0038] A transmission link selection module for the edge computing device to dynamically select a transmission link based on the network classification table and the network parameters;

[0039] An encoded video transmission module for the edge computing device to transmit the corresponding encoded videos based on the selected transmission link and transmission priority;

[0040] A bandwidth allocation dynamic adjustment module for, during the transmission process of the encoded video, the edge computing device to perform bandwidth prediction based on a pre-trained bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjust the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

[0041] Further, in the initialization module, the encoding table stores the correspondence between the encoding format and the network level; the encoding format at least includes H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP9, VP10, AV1, MPEG-4, AVC-Intra, and ProRes;

[0042] The network classification table stores the correspondence between the network level and the network parameters;

[0043] The video classification table stores the correspondence between the transmission priority and the video content, user account, and playback time limit;

[0044] The transmission link at least includes a wired transmission link, a wireless local area network, a cellular mobile network, Bluetooth, and satellite communication;

[0045] The network parameters at least include network bandwidth, network delay, jitter, packet loss rate, throughput, and TCP connection establishment time.

[0046] Further, the parallel encoding module is specifically used for:

[0047] The edge computing device obtains the original video to be transmitted, determines the network level of each parameter link based on the network classification table and the network parameters, matches the corresponding encoding format from the encoding table based on the optimal network level, and performs parallel encoding on each original video through a plurality of encoders to obtain the corresponding encoded video. During the encoding process, the bit rate is dynamically adjusted by combining the ROI encoding enhancement technology;

[0048] The transmission priority matching module is specifically used for:

[0049] The edge computing device obtains the video content, user account, and playback time limit of each encoded video, matches the transmission priority of each encoded video from the video classification table based on the video content, user account, and playback time limit, and stores each encoded video in the transmission queue in sequence based on the transmission priority.

[0050] Further, the transmission link selection module is specifically used for:

[0051] The edge computing device matches the network level corresponding to each transmission link from the network classification table based on the network parameters, dynamically selects the transmission link in the working state based on the network level, and dynamically adjusts the frame rate of the encoded video;

[0052] The encoded video transmission module is specifically used for:

[0053] The edge computing device selects the encoded video to be transmitted from the transmission queue based on the transmission priority, and transmits the selected encoded video through the selected transmission link. The encoded video is pre-cached for 500 ms before transmission.

[0054] Further, the bandwidth allocation dynamic adjustment module is specifically configured to:

[0055] Create a bandwidth prediction model based on the ARIMA network and the LSTM network, set the loss function of the bandwidth prediction model as the mean square error function, and construct a data set based on a large number of historical network parameters; the ARIMA network is used for short-term bandwidth prediction; the LSTM network is used for long-term bandwidth prediction;

[0056] Divide the data set into a training set, a validation set, and a test set based on a preset segmentation ratio. Train the bandwidth prediction model through the training set until the loss value of the loss function is less than a preset loss threshold. During the training process, compress the bandwidth prediction model through knowledge distillation technology and dynamic pruning technology; verify the trained bandwidth prediction model through the validation set, and determine whether the prediction accuracy is greater than a preset accuracy threshold. If not, the verification fails, expand the training set and continue training. If so, the verification is successful; test the bandwidth prediction model that has passed the verification through the test set, and determine whether the confidence level is greater than a preset confidence level threshold. If not, the test fails, expand the training set and continue training. If so, the test is successful, end the training, and deploy the bandwidth prediction model to the edge computing device;

[0057] During the transmission of the encoded video, the edge computing device performs bandwidth prediction based on the bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

[0058] The advantages of the present invention are:

[0059] 1. Create an encoding table, a network classification table, and a video classification table on edge computing devices of router, gateway, IPC, or ONU type, and monitor the current network parameters of each transmission link in real time. The edge computing device obtains the original videos to be transmitted, performs parallel encoding on each original video based on the encoding table, the network classification table, and the network parameters to obtain the corresponding encoded videos, matches the transmission priorities of the encoded videos based on the video classification table, dynamically selects the transmission link based on the network classification table and the network parameters, and transmits the corresponding encoded videos based on the selected transmission link and the transmission priority. During the transmission of the encoded videos, the edge computing device performs bandwidth prediction based on a pre-trained bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result. That is, before video transmission, first rate the network quality of each transmission link based on the network classification table and the network parameters to obtain the corresponding network level, match the corresponding encoding format from the encoding table for different network levels to encode the original videos, combine parallel encoding during the encoding process to improve the encoding efficiency, then match the transmission priorities of the encoded videos through the video classification table, dynamically select the optimal transmission link based on the current network parameters, combine the transmission priority and the optimal transmission link to transmit the encoded videos, dynamically adjust the bandwidth allocation of the network bandwidth during the transmission process, that is, transmit the most urgent encoded videos through the optimal transmission link, and the encoding format of the encoded videos is dynamically adjusted based on the transmission link to balance the transmission delay and the video quality, and dynamically adjust the allocated bandwidth, ultimately greatly improving the quality and stability of video transmission.

[0060] 2. By creating an encoding table storing the correspondence between the encoding format and the network level, creating a network classification table storing the correspondence between the network level and the network parameters, and creating a video classification table storing the correspondence between the transmission priority and the video content, user account, and playback time limit, the current network level of each transmission link, the transmission priority of the original videos, and the encoding format suitable for the current conditions can be grasped in real time in the future, so as to flexibly adjust the relevant parameters of the original videos, thereby improving the quality and stability of video transmission.

[0061] 3. Dynamically adjust the bit rate by combining the ROI encoding enhancement technology during the encoding process, that is, preferentially encode and enhance the quality of specific regions (such as faces, important objects, etc.) in the original video, so as to improve the overall visual quality under a limited bit rate, further improving the quality of video transmission.

[0062] 4. Match the network level corresponding to each transmission link from the network classification table based on network parameters, dynamically select the transmission link in the working state based on the network level, and dynamically adjust the frame rate of the encoded video, that is, dynamically select the transmission link with the optimal current network quality to transmit the encoded video. If the transmission link with the optimal current network quality is still poor before, appropriately reduce the frame rate of the encoded video, that is, flexibly select the transmission link and adjust the frame rate during the transmission of the encoded video to ensure the user's viewing experience.

[0063] 5. By pre-caching the encoded video for 500 ms before transmission, it is possible to ensure that there is no lag when switching the transmission link, that is, smoothly switch the transmission link, further improving the user's viewing experience.

[0064] 6. Construct a bandwidth prediction model through an ARIMA network for short-term bandwidth prediction and an LSTM network for long-term bandwidth prediction. The loss function of the bandwidth prediction model uses the mean square error function, which is suitable for regression problems such as time series prediction. It can impose a higher penalty on larger errors, thereby prompting the model to fit the data more accurately and effectively improving the accuracy of bandwidth prediction.

[0065] 7. Train the bandwidth prediction model with the training set until the loss value of the loss function is less than the preset loss threshold. During the training process, compress the bandwidth prediction model through knowledge distillation technology and dynamic pruning technology; calculate the prediction accuracy through the validation set to verify the trained bandwidth prediction model, and calculate the confidence through the test set to test the bandwidth prediction model that has passed the verification. If the test is successful, deploy the bandwidth prediction model to the edge computing device; that is, continuously compress, verify, and test during the training process of the bandwidth prediction model to effectively balance the model volume and prediction accuracy of the bandwidth prediction model, so as to better deploy it to the edge computing device. Description of the Drawings

[0066] The following further describes the present invention with reference to the drawings in conjunction with embodiments.

[0067] Figure 1 It is a flowchart of a video transmission method for routers, gateways, IP cameras, and ONUs according to the present invention.

[0068] Figure 2 It is a schematic structural diagram of a video transmission system for routers, gateways, IP cameras, and ONUs according to the present invention. Detailed Embodiments

[0069] The overall idea of the technical solution in the embodiments of this application is as follows: Before video transmission, first rate the network quality of each transmission link based on the network grading table and network parameters to obtain the corresponding network level. For different network levels, match the corresponding coding format from the coding table to encode the original video. During the encoding process, parallel encoding is combined to improve the encoding efficiency. Then, match the transmission priority of each encoded video through the video grading table, dynamically select the optimal transmission link based on the current network parameters, and transmit the encoded video by combining the transmission priority and the optimal transmission link. During the transmission process, dynamically adjust the bandwidth allocation of the network bandwidth to improve the quality and stability of video transmission.

[0070] Please refer to Figures 1 to 2 As shown, a preferred embodiment of a video transmission method for routers, gateways, IP cameras, and ONUs according to the present invention includes the following steps:

[0071] Step S1: An edge computing device with a device type of router, gateway, IP camera, or ONU creates a coding table, a network grading table, and a video grading table, and real-time monitors the current network parameters of each transmission link;

[0072] Step S2: The edge computing device obtains the original video to be transmitted, and performs parallel encoding on each original video based on the coding table, network grading table, and network parameters to obtain the corresponding encoded video;

[0073] Step S3: The edge computing device matches the transmission priority of each encoded video based on the video grading table;

[0074] Step S4: The edge computing device dynamically selects a transmission link based on the network grading table and network parameters;

[0075] Step S5: The edge computing device transmits the corresponding encoded video based on the selected transmission link and transmission priority;

[0076] Step S6: During the transmission of the encoded video, the edge computing device performs bandwidth prediction based on a pre-trained bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

[0077] By dynamically adjusting the coding format, bit rate, frame rate, bandwidth allocation of the network bandwidth, and transmission link, the network delay of video transmission can be effectively reduced.

[0078] In step S1, the coding table stores the corresponding relationship between the coding format and the network level; the coding format includes at least H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP9, VP10, AV1, MPEG-4, AVC-Intra, and ProRes;

[0079] The network classification table stores the corresponding relationship between network levels and network parameters;

[0080] The video classification table stores the corresponding relationship between transmission priorities and video content, user accounts, and play time limits;

[0081] The transmission link at least includes a wired transmission link, a wireless local area network, a cellular mobile network, Bluetooth, and satellite communication;

[0082] The network parameters at least include network bandwidth, network delay, jitter, packet loss rate, throughput, and TCP connection establishment time.

[0083] By creating an encoding table that stores the corresponding relationship between encoding formats and network levels, creating a network classification table that stores the corresponding relationship between network levels and network parameters, and creating a video classification table that stores the corresponding relationship between transmission priorities and video content, user accounts, and play time limits, it is possible to subsequently grasp in real time the current network level of each transmission link, the transmission priority of the original video, and the encoding format suitable for the current conditions, so as to flexibly adjust the relevant parameters of the original video, thereby improving the quality and stability of video transmission.

[0084] During specific implementation, video parameters can be dynamically adjusted based on network parameters. For example:

[0085] When the packet loss rate > 5% and lasts for 200 ms, trigger FEC redundancy adjustment (Reed - Solomon(10,8));

[0086] When the network delay > 150 ms, start dynamic frame rate adjustment (30 fps → 24 fps).

[0087] The specific content of step S2 is as follows:

[0088] The edge computing device obtains the original video to be transmitted, determines the network levels of each parameter link based on the network classification table and network parameters, matches the corresponding encoding format from the encoding table based on the optimal network level, and parallelly encodes each original video through a plurality of encoders to obtain the corresponding encoded video. During the encoding process, the bit rate is dynamically adjusted in combination with the ROI encoding enhancement technology (Region of Interest Encoding);

[0089] During the encoding process, the bitrate is dynamically adjusted by combining the ROI encoding enhancement technology. That is, by preferentially encoding and enhancing the quality of specific regions (such as faces, important objects, etc.) in the original video, the overall visual quality is improved under a limited bitrate, further enhancing the quality of video transmission. The specific operation steps of the ROI encoding enhancement technology are as follows: 1. Determine the region of interest: Identify the region of interest in the video through AI algorithms or other detection methods; 2. Adjust the encoding parameters: During the encoding process, adjust the quantization parameters according to the ROI region to optimize the bitrate allocation.

[0090] The specific steps of step S3 are as follows:

[0091] The edge computing device obtains the video content, user account, and playback time limit of each encoded video, matches the transmission priority of each encoded video from the video grading table based on the video content, user account, and playback time limit, and stores each encoded video in the transmission queue in sequence based on the transmission priority.

[0092] The specific steps of step S4 are as follows:

[0093] The edge computing device matches the network level corresponding to each transmission link from the network grading table based on the network parameters, dynamically selects the transmission link in the working state based on the network level, and dynamically adjusts the frame rate of the encoded video;

[0094] Match the network level corresponding to each transmission link from the network grading table through network parameters, dynamically select the transmission link in the working state based on the network level, and dynamically adjust the frame rate of the encoded video. That is, dynamically select the transmission link with the optimal current network quality for transmitting the encoded video. If the transmission link with the optimal current network quality is still poor before, appropriately reduce the frame rate of the encoded video. That is, flexibly select the transmission link and adjust the frame rate during the transmission of the encoded video to ensure the user's viewing experience.

[0095] The specific steps of step S5 are as follows:

[0096] The edge computing device selects the encoded video to be transmitted from the transmission queue based on the transmission priority, and transmits the selected encoded video through the selected transmission link, and pre-caches 500 ms of the encoded video before transmission.

[0097] By pre-caching 500 ms of the encoded video before transmission, it can ensure that there is no stuttering when switching the transmission link, that is, smoothly switch the transmission link, further enhancing the user's viewing experience.

[0098] The specific steps of step S6 are as follows:

[0099] Create a bandwidth prediction model based on the ARIMA network and the LSTM network. Set the loss function of the bandwidth prediction model as the mean square error function, and construct a data set based on a large number of historical network parameters. The ARIMA network is used for short-term bandwidth prediction, and the LSTM network is used for long-term bandwidth prediction.

[0100] Construct a bandwidth prediction model through the ARIMA network for short-term bandwidth prediction and the LSTM network for long-term bandwidth prediction. The loss function of the bandwidth prediction model uses the mean square error function, which is suitable for regression problems such as time series prediction. It can impose higher penalties on larger errors, thereby prompting the model to fit the data more accurately and effectively improving the accuracy of bandwidth prediction.

[0101] Divide the data set into a training set, a validation set, and a test set based on a preset splitting ratio. Train the bandwidth prediction model through the training set until the loss value of the loss function is less than the preset loss threshold. During the training process, compress the bandwidth prediction model through knowledge distillation technology and dynamic pruning technology. Verify the trained bandwidth prediction model through the validation set to determine whether the prediction accuracy is greater than the preset accuracy threshold. If not, the verification fails, and the training set is expanded and training continues. If so, the verification is successful. Test the bandwidth prediction model that has passed the verification through the test set to determine whether the confidence level is greater than the preset confidence threshold. If not, the test fails, and the training set is expanded and training continues. If so, the test is successful, end the training, and deploy the bandwidth prediction model to the edge computing device.

[0102] During the encoding video transmission process, the edge computing device performs bandwidth prediction based on the bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result. Specifically, in implementation, a sliding window algorithm (window size 50ms) can be combined for real-time bandwidth estimation.

[0103] Train the bandwidth prediction model through the training set until the loss value of the loss function is less than the preset loss threshold. During the training process, compress the bandwidth prediction model through knowledge distillation technology and dynamic pruning technology. Verify the trained bandwidth prediction model by calculating the prediction accuracy through the validation set, and test the bandwidth prediction model that has passed the verification by calculating the confidence level through the test set. If the test is successful, deploy the bandwidth prediction model to the edge computing device. That is, during the training process of the bandwidth prediction model, compression, verification, and testing are continuously performed to effectively balance the model volume and prediction accuracy of the bandwidth prediction model, so as to better deploy it on the edge computing device.

[0104] A preferred embodiment of a video transmission system for routers, gateways, IP cameras, and ONUs according to the present invention includes the following modules:

[0105] An initialization module is used to create a coding table, a network classification table, and a video classification table for edge computing devices of router, gateway, IPC, or ONU types, and to monitor the current network parameters of each transmission link in real time.

[0106] A parallel coding module is used for the edge computing device to obtain the original videos to be transmitted, and perform parallel coding on each original video based on the coding table, the network classification table, and the network parameters to obtain the corresponding coded videos.

[0107] A transmission priority matching module is used for the edge computing device to match the transmission priorities of each coded video based on the video classification table.

[0108] A transmission link selection module is used for the edge computing device to dynamically select a transmission link based on the network classification table and the network parameters.

[0109] A coded video transmission module is used for the edge computing device to transmit the corresponding coded videos based on the selected transmission link and the transmission priority.

[0110] A bandwidth allocation dynamic adjustment module is used during the transmission of the coded videos. The edge computing device performs bandwidth prediction based on a pre-trained bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

[0111] By dynamically adjusting the coding format, bit rate, frame rate, allocation of network bandwidth, and transmission link, the network delay of video transmission can be effectively reduced.

[0112] In the initialization module, the coding table stores the correspondence between the coding format and the network level; the coding format includes at least H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP9, VP10, AV1, MPEG-4, AVC-Intra, and ProRes.

[0113] The network classification table stores the correspondence between the network level and the network parameters.

[0114] The video classification table stores the correspondence between the transmission priority and the video content, user account, and playback time limit.

[0115] The transmission link includes at least a wired transmission link, a wireless local area network, a cellular mobile network, Bluetooth, and satellite communication.

[0116] The network parameters include at least network bandwidth, network delay, jitter, packet loss rate, throughput, and TCP connection establishment time.

[0117] By creating a coding table storing the correspondence between coding formats and network levels, creating a network classification table storing the correspondence between network levels and network parameters, and creating a video classification table storing the correspondence between transmission priorities and video content, user accounts, and play time limits, it is possible to subsequently monitor in real time the current network level of each transmission link, the transmission priority of the original video, and the coding format suitable for the current conditions, so as to flexibly adjust the relevant parameters of the original video, thereby improving the quality and stability of video transmission.

[0118] During specific implementation, video parameters can be dynamically adjusted based on network parameters. For example:

[0119] When the packet loss rate > 5% lasts for 200 ms, trigger FEC redundancy adjustment (Reed - Solomon(10,8));

[0120] When the network delay > 150 ms, start dynamic frame rate adjustment (30 fps → 24 fps).

[0121] The parallel coding module is specifically used for:

[0122] The edge computing device obtains the original video to be transmitted, determines the network level of each parameter link based on the network classification table and network parameters, matches the corresponding coding format from the coding table based on the optimal network level, and parallelly encodes each original video through a number of encoders to obtain the corresponding encoded video. During the encoding process, the bit rate is dynamically adjusted in combination with the ROI encoding enhancement technology (Region of Interest Encoding);

[0123] By dynamically adjusting the bit rate in combination with the ROI encoding enhancement technology during the encoding process, that is, by preferentially encoding and enhancing the quality of specific regions (such as faces, important objects, etc.) in the original video, the overall visual quality is improved under a limited bit rate, further improving the quality of video transmission. The specific operation steps of the ROI encoding enhancement technology are as follows: 1. Determine the region of interest: Identify the region of interest in the video through AI algorithms or other detection methods; 2. Adjust the encoding parameters: During the encoding process, adjust the quantization parameters according to the ROI region to optimize the bit rate allocation.

[0124] The transmission priority matching module is specifically used for:

[0125] The edge computing device obtains the video content, user account, and play time limit of each encoded video, matches the transmission priority of each encoded video from the video classification table based on the video content, user account, and play time limit, and stores each encoded video in the transmission queue in sequence based on the transmission priority.

[0126] The transmission link selection module is specifically used for:

[0127] The edge computing device matches the network levels corresponding to each transmission link from the network classification table based on the network parameters, dynamically selects the transmission links in the working state based on the network levels, and dynamically adjusts the frame rate of the encoded video;

[0128] Match the network levels corresponding to each transmission link from the network classification table through the network parameters, dynamically select the transmission links in the working state based on the network levels, and dynamically adjust the frame rate of the encoded video, that is, dynamically select the transmission link with the optimal current network quality for transmitting the encoded video. If the transmission link with the optimal current network quality is still poor before, appropriately reduce the frame rate of the encoded video. That is, flexibly select the transmission link and adjust the frame rate during the transmission of the encoded video to ensure the viewing experience of users.

[0129] The encoded video transmission module is specifically used for:

[0130] The edge computing device selects the encoded video to be transmitted from the transmission queue based on the transmission priority, transmits the encoded video to be transmitted through the selected transmission link, and pre-caches the encoded video for 500 ms before transmission.

[0131] By pre-caching the encoded video for 500 ms before transmission, it can ensure that there is no lag when switching the transmission link, that is, smoothly switch the transmission link, and further improve the viewing experience of users.

[0132] The bandwidth allocation dynamic adjustment module is specifically used for:

[0133] Create a bandwidth prediction model based on the ARIMA network and the LSTM network, set the loss function of the bandwidth prediction model as the mean square error function, and construct a data set based on a large number of historical network parameters; the ARIMA network is used for short-term bandwidth prediction; the LSTM network is used for long-term bandwidth prediction;

[0134] Construct a bandwidth prediction model through the ARIMA network for short-term bandwidth prediction and the LSTM network for long-term bandwidth prediction, and the loss function of the bandwidth prediction model uses the mean square error function, which is suitable for regression problems such as time series prediction. It can impose higher penalties on larger errors, thereby prompting the model to fit the data more accurately and effectively improving the accuracy of bandwidth prediction.

[0135] Divide the dataset into a training set, a validation set, and a test set based on a preset segmentation ratio. Train the bandwidth prediction model using the training set until the loss value of the loss function is less than a preset loss threshold. During the training process, compress the bandwidth prediction model using knowledge distillation technology and dynamic pruning technology. Validate the trained bandwidth prediction model using the validation set to determine whether the prediction accuracy is greater than a preset accuracy threshold. If not, the validation fails, and the training set is expanded and training continues. If so, the validation succeeds. Test the bandwidth prediction model that has passed validation using the test set to determine whether the confidence level is greater than a preset confidence threshold. If not, the test fails, and the training set is expanded and training continues. If so, the test succeeds, training ends, and the bandwidth prediction model is deployed to an edge computing device.

[0136] During the transmission of the encoded video, the edge computing device performs bandwidth prediction based on the bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result. In specific implementation, a sliding window algorithm (window size 50ms) can be combined for real-time bandwidth estimation.

[0137] Train the bandwidth prediction model using the training set until the loss value of the loss function is less than a preset loss threshold. During the training process, compress the bandwidth prediction model using knowledge distillation technology and dynamic pruning technology. Calculate the prediction accuracy using the validation set to validate the trained bandwidth prediction model, and calculate the confidence level using the test set to test the bandwidth prediction model that has passed validation. If the test is successful, deploy the bandwidth prediction model to an edge computing device. That is, during the training process of the bandwidth prediction model, continuous compression, validation, and testing are performed to effectively balance the model size and prediction accuracy of the bandwidth prediction model, so as to better deploy it on the edge computing device.

[0138] In summary, the advantages of the present invention are as follows:

[0139] 1. By creating an encoding table, a network classification table, and a video classification table on edge computing devices of router, gateway, IPC, or ONU types, and monitoring the current network parameters of each transmission link in real time; the edge computing device obtains the original videos to be transmitted, performs parallel encoding on each original video based on the encoding table, the network classification table, and the network parameters to obtain the corresponding encoded videos, matches the transmission priorities of each encoded video based on the video classification table, dynamically selects the transmission link based on the network classification table and the network parameters, and transmits the corresponding encoded videos based on the selected transmission link and the transmission priority; during the transmission of the encoded videos, the edge computing device performs bandwidth prediction based on a pre-trained bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result; that is, before video transmission, first rate the network quality of each transmission link based on the network classification table and the network parameters to obtain the corresponding network level, match the corresponding encoding format from the encoding table for different network levels to encode the original videos, combine parallel encoding during the encoding process to improve the encoding efficiency, then match the transmission priorities of each encoded video through the video classification table, dynamically select the optimal transmission link based on the current network parameters, combine the transmission priority and the optimal transmission link to transmit the encoded videos, dynamically adjust the bandwidth allocation of the network bandwidth during the transmission process, that is, transmit the most urgent encoded videos through the optimal transmission link, and the encoding format of the encoded videos is dynamically adjusted based on the transmission link to balance the transmission delay and the video quality, and dynamically adjust the allocated bandwidth, ultimately greatly improving the quality and stability of video transmission.

[0140] 2. By creating an encoding table storing the correspondence between the encoding format and the network level, creating a network classification table storing the correspondence between the network level and the network parameters, and creating a video classification table storing the correspondence between the transmission priority and the video content, user account, and playback time limit, the current network level of each transmission link, the transmission priority of the original videos, and the encoding format suitable for the current conditions can be mastered in real time in the future, so as to flexibly adjust the relevant parameters of the original videos, thereby improving the quality and stability of video transmission.

[0141] 3. By dynamically adjusting the bit rate by combining the ROI encoding enhancement technology during the encoding process, that is, by preferentially encoding and enhancing the quality of specific regions (such as faces, important objects, etc.) in the original video, the overall visual quality is improved under a limited bit rate, further improving the quality of video transmission.

[0142] 4. Match the network levels corresponding to each transmission link from the network classification table through network parameters, dynamically select the transmission links in the working state based on the network levels, and dynamically adjust the frame rate of the encoded video, that is, dynamically select the transmission link with the optimal current network quality for transmitting the encoded video. If the transmission link with the optimal current network quality is still poor as before, appropriately reduce the frame rate of the encoded video, that is, flexibly select the transmission link and adjust the frame rate during the transmission of the encoded video to ensure the user's viewing experience.

[0143] 5. By pre-caching the encoded video for 500 ms before transmission, it can be ensured that there will be no stuttering when switching the transmission link, that is, the transmission link can be switched smoothly, further improving the user's viewing experience.

[0144] 6. Construct a bandwidth prediction model through the ARIMA network for short-term bandwidth prediction and the LSTM network for long-term bandwidth prediction, and the loss function of the bandwidth prediction model uses the mean square error function, which is suitable for regression problems such as time series prediction. It can impose higher penalties on larger errors, thus prompting the model to fit the data more accurately and effectively improving the accuracy of bandwidth prediction.

[0145] 7. Train the bandwidth prediction model with the training set until the loss value of the loss function is less than the preset loss threshold. During the training process, compress the bandwidth prediction model through knowledge distillation technology and dynamic pruning technology; calculate the prediction accuracy through the validation set to verify the trained bandwidth prediction model, calculate the confidence through the test set to test the bandwidth prediction model that has passed the verification, and deploy the bandwidth prediction model to the edge computing device if the test is successful; that is, continuously compress, verify and test during the training process of the bandwidth prediction model to effectively balance the model volume and prediction accuracy of the bandwidth prediction model, so as to better deploy it to the edge computing device.

[0146] Although the specific implementation manners of the present invention have been described above, those skilled in the art should understand that the specific embodiments we described are illustrative rather than used to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered by the scope protected by the claims of the present invention.

Claims

1. A video transmission method for routers, gateways, IPCs, and ONUs, characterized in that: The steps include: Step S1: The edge computing device whose device type is a router, gateway, IPC or ONU creates a coding table, a network classification table, and a video classification table, and monitors the current network parameters of each transmission link in real time; Step S2: The edge computing device obtains the original video to be transmitted, and encodes each original video in parallel based on the encoding table, the network classification table, and the network parameters to obtain a corresponding encoded video; Step S3: The edge computing device matches the transmission priority of each encoded video based on the video rating table; Step S4: The edge computing device dynamically selects a transmission link based on the network classification table and network parameters; Step S5: The edge computing device transmits the corresponding encoded video based on the selected transmission link and transmission priority; Step S6: During the encoded video transmission process, the edge computing device performs bandwidth prediction based on the pre-trained bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

2. A video transmission method for a router, a gateway, an IPC, or an ONU as claimed in claim 1, characterized in that: In step S1, the encoding table stores a correspondence between encoding formats and network levels; the encoding formats include at least H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP9, ​​VP10, AV1, MPEG-4, AVC-Intra and ProRes; The network classification table stores the corresponding relationship between network levels and network parameters; The video rating table stores the corresponding relationship between transmission priority and video content, user account, and playback time limit; The transmission link at least includes a wired transmission link, a wireless local area network, a cellular mobile network, Bluetooth and satellite communication; The network parameters include at least network bandwidth, network delay, jitter, packet loss rate, throughput and TCP connection establishment time.

3. A video transmission method for a router, a gateway, an IPC, or an ONU as claimed in claim 1, characterized in that: The step S2 is specifically as follows: The edge computing device obtains the original video to be transmitted, determines the network level of each parameter link based on the network classification table and the network parameters, matches the corresponding encoding format from the encoding table based on the optimal network level, and encodes each original video in parallel through a plurality of encoders to obtain a corresponding encoded video. During the encoding process, the bit rate is dynamically adjusted in combination with the ROI encoding enhancement technology; The step S3 is specifically as follows: The edge computing device obtains the video content, user account and playback time limit of each encoded video, matches the transmission priority of each encoded video from the video rating table based on the video content, user account and playback time limit, and stores each encoded video in sequence in a transmission queue based on the transmission priority.

4. A video transmission method for a router, a gateway, an IPC, or an ONU as claimed in claim 1, characterized in that: The step S4 is specifically as follows: The edge computing device matches the network level corresponding to each transmission link from the network classification table based on the network parameters, dynamically selects the transmission link in working state based on the network level, and dynamically adjusts the frame rate of the encoded video; The step S5 is specifically as follows: The edge computing device selects the encoded video to be transmitted from the transmission queue based on the transmission priority, transmits the encoded video to be transmitted through the selected transmission link, and pre-caches 500ms of the encoded video before transmission.

5. A video transmission method for a router, a gateway, an IPC, or an ONU as claimed in claim 1, characterized in that: The step S6 is specifically as follows: A bandwidth prediction model is created based on the ARIMA network and the LSTM network, the loss function of the bandwidth prediction model is set to the mean square error function, and a data set is constructed based on a large number of historical network parameters; the ARIMA network is used for short-term bandwidth prediction; the LSTM network is used for long-term bandwidth prediction; Based on a preset segmentation ratio, the data set is divided into a training set, a validation set and a test set. The bandwidth prediction model is trained through the training set until the loss value of the loss function is less than a preset loss threshold. During the training process, the bandwidth prediction model is compressed through knowledge distillation technology and dynamic pruning technology; the trained bandwidth prediction model is verified through the validation set to determine whether the prediction accuracy is greater than the preset accuracy threshold. If not, the verification fails, and the training set is expanded to continue training. If so, the verification succeeds; the successfully verified bandwidth prediction model is tested through the test set to determine whether the confidence is greater than the preset confidence threshold. If not, the test fails, and the training set is expanded to continue training. If so, the test succeeds, the training ends, and the bandwidth prediction model is deployed to the edge computing device; During the encoded video transmission process, the edge computing device performs bandwidth prediction based on the bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

6. A video transmission system for routers, gateways, IPCs, and ONUs, characterized in that: Includes the following modules: An initialization module is used to create a coding table, a network classification table, and a video classification table for edge computing devices whose device types are routers, gateways, IPCs, or ONUs, and to monitor the current network parameters of each transmission link in real time; A parallel encoding module is used for the edge computing device to obtain the original video to be transmitted, and to perform parallel encoding on each original video based on the encoding table, the network classification table and the network parameters to obtain a corresponding encoded video; A transmission priority matching module, used for the edge computing device to match the transmission priority of each encoded video based on the video rating table; A transmission link selection module, used for the edge computing device to dynamically select a transmission link based on the network classification table and network parameters; A coded video transmission module, used for the edge computing device to transmit the corresponding coded video based on the selected transmission link and transmission priority; The bandwidth allocation dynamic adjustment module is used in the encoded video transmission process, where the edge computing device performs bandwidth prediction based on a pre-trained bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

7. A video transmission system for routers, gateways, IPCs, and ONUs as claimed in claim 6, characterized in that: In the initialization module, the encoding table stores a correspondence between encoding formats and network levels; the encoding formats include at least H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP9, ​​VP10, AV1, MPEG-4, AVC-Intra and ProRes; The network classification table stores the corresponding relationship between network levels and network parameters; The video rating table stores the corresponding relationship between transmission priority and video content, user account, and playback time limit; The transmission link at least includes a wired transmission link, a wireless local area network, a cellular mobile network, Bluetooth and satellite communication; The network parameters include at least network bandwidth, network delay, jitter, packet loss rate, throughput and TCP connection establishment time.

8. A video transmission system for routers, gateways, IPCs, and ONUs as claimed in claim 6, characterized in that: The parallel encoding module is specifically used for: The edge computing device obtains the original video to be transmitted, determines the network level of each parameter link based on the network classification table and the network parameters, matches the corresponding encoding format from the encoding table based on the optimal network level, and encodes each original video in parallel through a plurality of encoders to obtain a corresponding encoded video. During the encoding process, the bit rate is dynamically adjusted in combination with the ROI encoding enhancement technology; The transmission priority matching module is specifically used for: The edge computing device obtains the video content, user account and playback time limit of each encoded video, matches the transmission priority of each encoded video from the video rating table based on the video content, user account and playback time limit, and stores each encoded video in sequence in a transmission queue based on the transmission priority.

9. A video transmission system for routers, gateways, IPCs, and ONUs as claimed in claim 6, characterized in that: The transmission link selection module is specifically used for: The edge computing device matches the network level corresponding to each transmission link from the network classification table based on the network parameters, dynamically selects the transmission link in working state based on the network level, and dynamically adjusts the frame rate of the encoded video; The coded video transmission module is specifically used for: The edge computing device selects the encoded video to be transmitted from the transmission queue based on the transmission priority, transmits the encoded video to be transmitted through the selected transmission link, and pre-caches 500ms of the encoded video before transmission.

10. A video transmission system for routers, gateways, IPCs, and ONUs as claimed in claim 6, characterized in that: The bandwidth allocation dynamic adjustment module is specifically used for: A bandwidth prediction model is created based on the ARIMA network and the LSTM network, the loss function of the bandwidth prediction model is set to the mean square error function, and a data set is constructed based on a large number of historical network parameters; the ARIMA network is used for short-term bandwidth prediction; the LSTM network is used for long-term bandwidth prediction; Based on a preset segmentation ratio, the data set is divided into a training set, a validation set and a test set. The bandwidth prediction model is trained through the training set until the loss value of the loss function is less than a preset loss threshold. During the training process, the bandwidth prediction model is compressed through knowledge distillation technology and dynamic pruning technology; the trained bandwidth prediction model is verified through the validation set to determine whether the prediction accuracy is greater than the preset accuracy threshold. If not, the verification fails, and the training set is expanded to continue training. If so, the verification succeeds; the successfully verified bandwidth prediction model is tested through the test set to determine whether the confidence is greater than the preset confidence threshold. If not, the test fails, and the training set is expanded to continue training. If so, the test succeeds, the training ends, and the bandwidth prediction model is deployed to the edge computing device; During the encoded video transmission process, the edge computing device performs bandwidth prediction based on the bandwidth prediction model to obtain a bandwidth prediction result, and dynamically adjusts the bandwidth allocation of the network bandwidth based on the bandwidth prediction result.

Images