Video Transmission Method and System Based on Scalable Video Super-Resolution Model

Abstract

The present invention discloses a video transmission method and system based on a scalable video super-resolution model. The method comprises the following steps: collecting network status information, and determining the model size and reconstruction resolution of the scalable video super-resolution model according to the network status information; after receiving a request signal, feeding back a corresponding low-resolution video block according to the request signal; reconstructing the low-resolution video block using the scalable video super-resolution model after determining the model size and reconstruction resolution to obtain a high-resolution video block; transmitting the high-resolution video block to a playback buffer, and then playing the high-resolution video block.

Description

Technical Field

[0001] The present invention relates to video super-resolution transmission, and in particular to a video transmission method and system based on a scalable video super-resolution model. Background Art

[0002] Delivering HD / UHD video services over backhaul or core networks with poor or fluctuating bandwidth is crucial but challenging. Traditional Dynamic Adaptive Streaming over HTTP (DASH) is highly dependent on network conditions and delivers poor video quality under poor network conditions.

[0003] To improve video quality, existing technologies use the following methods:

[0004] 1. Video super-resolution (VSR) is used to reconstruct low-resolution videos into high-resolution videos near the terminal, breaking network dependence. However, VSR requires powerful processing capabilities, and most existing VSR models cannot achieve line-speed VSR processing on mobile devices.

[0005] 2. Reconstruct low-resolution videos at edge nodes to save bandwidth from servers to edge nodes. The time complexity of the VSR model should be dynamically matched to the available computing power of edge nodes. However, since edge nodes are shared by different applications, the computing power of edge nodes is time-varying. At the same time, the bit rate of the high-resolution video reconstructed from the low-resolution video should match the dynamic access network conditions, otherwise additional processing delays or transmission delays will be introduced.

[0006] 3. DASH-based video transmission framework, which uses VSR on the client to improve video quality, but requires the client to have strong computing power, which is not practical for mobile devices;

[0007] To overcome the impact of network bandwidth on video quality, existing technologies have proposed an edge-assisted adaptive video streaming solution, which enhances the cached low-quality video blocks downloaded by the edge VSR from the remote server. Although this method can also provide high-quality video when backhaul network resources are scarce, it does not consider the dynamic state of available computing power on the edge node. When the available computing power cannot meet the requirements of the VSR, additional super-resolution processing delay will be introduced, which will lead to more rejections and reduce user QoE. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to improve video quality, and the purpose is to provide a video transmission method and system based on a scalable video super-resolution model, which solves the problem of poor video quality.

[0009] The present invention is achieved through the following technical solutions:

[0010] A first aspect provides a video transmission method based on a scalable video super-resolution model, comprising the following steps:

[0011] Collecting network status information, and determining a model size and a reconstruction resolution of a scalable video super-resolution model based on the network status information and available computing resources on the edge node;

[0012] After receiving the request signal, the server feeds back the corresponding low-resolution video block according to the request signal;

[0013] The server transmits the corresponding low-resolution video blocks to the edge nodes;

[0014] Reconstructing low-resolution video blocks on edge nodes using the scalable video super-resolution model after determining the model size and reconstruction resolution to obtain high-resolution video blocks;

[0015] After the high-resolution video block is transferred to the play buffer, the high-resolution video block is played.

[0016] Due to the different performance of various computing devices and the different real-time computing capabilities of edge nodes, for the decision to reconstruct each low-resolution video block, the model size and reconstruction resolution of the scalable video super-resolution model are first determined based on the network status information, and then the low-resolution video block is reconstructed using the determined scalable video super-resolution model. The low-resolution video block is reconstructed into a high-resolution video block on the edge node, which can provide users with high-quality video even when network resources are scarce. At the same time, the time-varying computing capabilities of the edge nodes are also taken into account, and the model size of the scalable video super-resolution model is designed to be scalable. The matching degree of the reconstructed high-resolution video block with the dynamic access network is improved, which reduces processing delay and transmission delay and improves video quality.

[0017] Furthermore, the specific steps of determining the model size and reconstruction resolution of the scalable video super-resolution model based on the above network status information are as follows:

[0018] Pre-configure a group of neural network models, and select a matching neural network model from the group of neural network models according to the number of residual convolution blocks;

[0019] The output path of the exit scalable video super-resolution model is determined based on the output features of different numbers of residual convolution blocks.

[0020] Different numbers of residual convolution blocks result in different model levels. The greater the number of residual convolution blocks, the higher the model level and the better the reconstructed video quality, but the corresponding reconstruction time is longer. A set of neural network models is pre-configured for each possible low-resolution reconstruction, so that the neural network model that best matches the computing device can be selected on the edge node.

[0021] To further adapt to the time-varying computing power of edge nodes, it is necessary to dynamically determine the output path of the scalable video super-resolution model through the output features of different numbers of residual convolution blocks. The more residual convolution blocks added to the output path, the better the reconstructed video quality. Even if the number of residual convolution blocks in the output path is small, the user experience quality is still greatly improved compared to directly watching low-resolution video blocks.

[0022] Furthermore, the low-resolution video blocks are reconstructed using the scalable video super-resolution model after determining the model size and reconstruction resolution. The specific steps are as follows:

[0023] caching the low-resolution video blocks into a low-resolution video block buffer, and sequentially reconstructing the low-resolution video blocks in the low-resolution video block buffer according to a first-in-first-out principle to obtain high-resolution video blocks;

[0024] The high-resolution video chunks are cached in the high-resolution video chunk buffer; for the i-th low-resolution video chunk in the download buffer, the DRL agent should determine the video reconstruction resolution level (r i ,r i ′), r i ,r i ′∈R;

[0025] set up represents all reconstruction pairs; let represents the maximum length of the super-resolution buffer, and Indicates the start time of reconstructing the i-th low-resolution video block;

[0026] Calculate the reconstruction start time of the i-th low-resolution video block according to formula (1);

[0027] For the i-th low-resolution video chunk in the low-resolution video chunk buffer, the Deep Reinforcement Learning (DRL) agent should determine the video reconstruction resolution level (r i ,r′ i ), r i ,r′ i ∈R;

[0028] set up represents all reconstruction pairs; let represents the maximum length of the super-resolution buffer, and represents the reconstruction start time of the i-th low-resolution video block;

[0029]

[0030] in, represents the reconstruction start time of the i-th low-resolution video block; represents the reconstruction start time of the (i-1)th low-resolution video block; φ i-1 (r i-1 ,r' i-1 ) represents the reconstruction processing time for reconstructing the (i-1)th low-resolution video block into a high-resolution video block; Indicates the The cache start time of the high-resolution video block cached to the playback buffer, D represents all the reconstruction pairs, Indicates the maximum length of the high-resolution video block buffer;

[0031] The reconstruction processing time of the i-th low-resolution video block is calculated according to formula (2):

[0032]

[0033] Among them, φ i (r i ,r' i ) represents the reconstruction processing time of the i-th low-resolution video block; u i represents the computing power available on the edge node to reconstruct the i-th low-resolution video block; Indicates the baseline reconstruction time.

[0034] Furthermore, the caching start time of caching the high-resolution video block to the playback buffer is: after the i-th low-resolution video block is reconstructed and the (i-1)-th high-resolution video block is transferred to the playback buffer, the i-th high-resolution video block starts to be transferred to the playback buffer;

[0035] According to formula (3), the cache start time of the i-th high-resolution video block cached to the playback buffer is calculated.

[0036]

[0037] in, Indicates the cache start time of the i-th high-resolution video block cached to the playback buffer; Indicates the cache start time of the (i-1)th high-resolution video block cached to the playback buffer; d i-1 represents the download time of (i-1) high-resolution video chunks; represents the reconstruction start time of the i-th low-resolution video block; φ i (r i ,r' i ) represents the reconstruction processing time of reconstructing the i-th low-resolution video block into a high-resolution video block; Indicates the The start time of the high-resolution video chunks. Indicates the maximum length of the playback buffer.

[0038] Furthermore, the playback time of the above high-resolution video block is: the playback of the (i-1)th high-resolution video block is completed, and the i-th high-resolution video block has been cached in the playback buffer;

[0039] Calculate the playback start time of the i-th high-resolution video block according to formula (4):

[0040]

[0041] Among them, t i represents the start time of playback of the i-th high-resolution video block; t i-1 represents the playback start time of the (i-1)th high-resolution video block; T represents the time length of the high-resolution video block or the low-resolution video block; represents the cache start time of the i-th high-resolution video block cached to the playback buffer; d i represents the download time of the i-th high-resolution video chunk.

[0042] Furthermore, when playing the high-resolution video blocks, if the high-resolution video blocks are exhausted in the playback buffer, the high-resolution video blocks need to be re-cached;

[0043] If the high-resolution video block needs to be re-cached, the time for re-caching the high-resolution video block is calculated according to formula (5):

[0044] τ i =t i -(t i-1 +T) Formula (5)

[0045] Among them, τ i represents the time to re-cache the i-th high-resolution video chunk; t i represents the start time of playback of the i-th high-resolution video block; t i-1 represents the playback start time of the i-1th high-resolution video block; T represents the time length of the high-resolution video block or the low-resolution video block.

[0046] A second aspect provides a video transmission system based on a scalable video super-resolution model. The video transmission system is used to implement the above-mentioned video transmission method based on the scalable video super-resolution model. The video transmission system includes:

[0047] Client, used to send request signals and receive high-resolution video chunks;

[0048] A server for receiving a request signal and sending a low-resolution video block;

[0049] edge nodes,

[0050] The edge node is connected to the client for communication, and is configured to receive a request signal sent by the client and send the request signal to the server;

[0051] The edge node is communicatively connected to the server, and is configured to receive the low-resolution video blocks sent by the server, reconstruct the low-resolution video blocks to obtain high-resolution video blocks, and send the high-resolution video blocks to the client.

[0052] The above-mentioned edge nodes can collect network status information, have the ability to support scalable video super-resolution processing, and can adaptively adjust the model size and reconstruction resolution of the scalable video super-resolution model, integrating edge computing and video super-resolution to improve video quality.

[0053] Furthermore, the server is used to store the low-resolution video, and divide the low-resolution video into N low-resolution video blocks with the same time length T, to obtain a low-resolution video block group consisting of the N low-resolution video blocks.

[0054] Furthermore, the edge node includes a monitor, a DRL agent, and a Scalable Video Super Resolution (SVSR) processor;

[0055] The monitor is in communication with the client and the DRL agent and the server, and is used to monitor the network status of the server, the edge node and the client;

[0056] The DRL agent is used to receive network status information and determine the model size and reconstruction resolution of the scalable video super-resolution model according to the network status information;

[0057] The SVSR processor is in communication with the DRL agent, and is configured to reconstruct the low-resolution video block using the scalable video super-resolution model after determining the model size and reconstruction resolution to obtain a high-resolution video block.

[0058] Furthermore, the SVSR processor includes a low-resolution video block buffer, a high-resolution video block buffer, and a processing area;

[0059] The low-resolution video block buffer is used to cache the low-resolution video blocks sent by the server to the SVSR processor;

[0060] Reconstructing the low-resolution video blocks in the low-resolution video block buffer in the processing area in sequence according to a first-in-first-out principle to obtain high-resolution video blocks;

[0061] The high-resolution video block is cached in the high-resolution video block buffer.

[0062] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0063] Due to the varying performance of various computing devices and the varying real-time computing capabilities of edge nodes, the decision to reconstruct each low-resolution video block is based on the network status information to determine the model size and reconstruction resolution of the scalable video super-resolution model. The determined scalable video super-resolution model is then used to reconstruct the low-resolution video block. This reconstructs the low-resolution video block into a high-resolution video block on the edge node, providing users with high-quality video even when network resources are scarce. The time-varying computing capabilities of edge nodes are also taken into account, and the model size of the scalable video super-resolution model is designed to be scalable. This improves the compatibility of the reconstructed high-resolution video block with the dynamic access network, reducing processing and transmission delays and improving video quality. This solution also establishes a joint optimization for ultra-high-definition video transmission that integrates edge computing and video super-resolution. The system's performance is primarily reflected in the super-resolution process at the edge node and the transmission of the reconstructed video from the edge node to the client. To ensure smooth video streaming service, the super-resolution buffer and playback buffer must meet a series of conditions to avoid video interruptions. While ensuring the smoothness of video streaming services, an optimization model is developed to improve video transmission quality, reduce rebuffering time, and minimize quality fluctuations. The decision variables are the reconstructed video resolution level and the configuration of the SVSR model. A reinforcement learning approach is used to solve this joint optimization problem. Intelligent edge nodes supporting reinforcement learning are trained until the algorithm converges. The trained reinforcement learning algorithm is then used to make decisions about the reconstructed video resolution level and the SVSR model configuration, improving user QoE while conserving network bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the examples. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be considered as limiting the scope. A person of ordinary skill in the art can also derive other relevant drawings based on these drawings without inventive effort. In the drawings:

[0065] Figure 1 Flowchart provided for Example 1;

[0066] Figure 2 The scalable video super-resolution model structure provided in Example 1;

[0067] Figure 3 This is a system block diagram provided for Example 2.

[0068] Markings and corresponding parts names in the accompanying drawings:

[0069] 100 - client, 110 - playback buffer, 200 - edge node, 210 - monitor, 220 - DRL agent, 230 - SVSR processor, 231 - low-resolution video block buffer, 232 - processing area, 233 - high-resolution video block buffer, 300 - server. DETAILED DESCRIPTION

[0070] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples and drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0071] Example 1

[0072] like Figure 1 As shown, this embodiment 1 provides a video transmission method based on a scalable video super-resolution model, comprising the following steps:

[0073] S1. Collect network status information, and determine the model size and reconstruction resolution of the scalable video super-resolution model based on the network status information;

[0074] S2. After receiving the request signal, feeding back the corresponding low-resolution video block according to the request signal;

[0075] S3. Reconstruct the low-resolution video block using the scalable video super-resolution model after determining the model size and reconstruction resolution to obtain a high-resolution video block;

[0076] S4: After transferring the high-resolution video block to the playback buffer, play the high-resolution video block.

[0077] Due to the different performance of various computing devices and the different real-time computing capabilities of edge nodes, for the decision to reconstruct each low-resolution video block, the model size and reconstruction resolution of the scalable video super-resolution model are first determined based on the network status information, and then the low-resolution video block is reconstructed using the determined scalable video super-resolution model. The low-resolution video block is reconstructed into a high-resolution video block on the edge node, which can provide users with high-quality video even when network resources are scarce. At the same time, the time-varying computing capabilities of the edge nodes are also taken into account, and the model size of the scalable video super-resolution model is designed to be scalable. The matching degree of the reconstructed high-resolution video block with the dynamic access network is improved, which reduces processing delay and transmission delay and improves video quality.

[0078] Specific embodiments, such as Figure 2 As shown, the specific steps of determining the model size and reconstruction resolution of the scalable video super-resolution model based on the above network status information are as follows:

[0079] Pre-configure a group of neural network models, and select a matching neural network model from the group of neural network models according to the number of residual convolution blocks;

[0080] The output path of the exit scalable video super-resolution model is determined based on the output features of different numbers of residual convolution blocks.

[0081] Different numbers of residual convolution blocks result in different model levels. The greater the number of residual convolution blocks, the higher the model level and the better the reconstructed video quality, but the corresponding reconstruction time is longer. A set of neural network models is pre-configured for each possible low-resolution reconstruction, so that the neural network model that best matches the computing device can be selected on the edge node.

[0082] To further adapt to the time-varying computing power of edge nodes, it is necessary to generate multiple output paths for each video super-resolution neural network model. This means adding direct connections from intermediate blocks to the final layer in the scalable super-resolution model, essentially establishing an early exit mechanism. The more residual convolutional blocks added to the output path, the better the video quality after SVSR reconstruction, but at the cost of increased reconstruction time. Even with a small number of residual convolutional blocks in the output path, the user experience quality is still significantly improved compared to directly viewing low-resolution video.

[0083] In a specific embodiment, the SR processing starts at the time. At the intelligent edge node, the low-resolution video blocks downloaded from the server are first cached in the download buffer. Then, the SVSR processor reconstructs the low-resolution video blocks in the download buffer according to the first-in-first-out (FIFO) principle and outputs the high-resolution video blocks to the super-resolution buffer. For the i-th low-resolution video block in the download buffer, the DRL agent shall determine the video reconstruction resolution level (r i ,r i ′), r i ,r i ′∈R.

[0084] set up Denotes all reconstruction pairs. represents the maximum length of the super-resolution buffer, and Indicates the start time of reconstructing the i-th low-resolution video block.

[0085] The start time of reconstructing the i-th low-resolution video block can be defined as:

[0086]

[0087] in, represents the reconstruction start time of the i-th low-resolution video block; represents the reconstruction start time of the i-1th low-resolution video block; φ i-1 (r i-1 ,r' i-1 ) represents the reconstruction processing time for reconstructing the i-1th low-resolution video block into a high-resolution video block; Indicates the The cache start time of the high-resolution video block cached to the playback buffer, D represents all the reconstruction pairs, Indicates the maximum length of the high-resolution video block buffer;

[0088] According to formula (6), the computing power available on the edge node for reconstructing the δth low-resolution video block is calculated as follows:

[0089]

[0090] Among them, u δ represents the computing power available on the edge node to reconstruct the δth low-resolution video block; φ δ (r δ ,r' δ ) represents the reconstruction processing time of the δth low-resolution video block; represents the baseline reconstruction time, which is used to estimate the time required to reconstruct a low-resolution video block of time length T into a high-resolution video block;

[0091] By calculating u i-1 ,u i-2 ,…,u i-k To estimate the computational power u that can be used to reconstruct the i-th low-resolution video block i .

[0092] The reconstruction processing time of the i-th low-resolution video block is calculated according to formula (2):

[0093]

[0094] Among them, φ i (r i ,r' i ) represents the reconstruction processing time of the i-th low-resolution video block; u i represents the computing power available on the edge node 200 for reconstructing the i-th low-resolution video block; Indicates the baseline reconstruction time.

[0095] In a specific embodiment, the caching start time of caching the high-resolution video block to the playback buffer is: after the i-th low-resolution video block is reconstructed and the i-1-th high-resolution video block is transferred to the playback buffer, the i-th high-resolution video block starts to be transferred to the playback buffer;

[0096] According to formula (3), the cache start time of the i-th high-resolution video block cached to the playback buffer is calculated.

[0097]

[0098] in, Indicates the cache start time of the i-th high-resolution video block cached to the playback buffer; Indicates the cache start time of the i-1th high-resolution video block cached to the playback buffer; d i-1 represents the download time of (i-1) high-resolution video chunks; represents the reconstruction start time of the i-th low-resolution video block; φ i (r i ,r' i ) represents the reconstruction processing time of reconstructing the i-th low-resolution video block into a high-resolution video block; Indicates the The start time of the high-resolution video chunks. Indicates the maximum length of the playback buffer.

[0099] In a specific embodiment, the playback time of the high-resolution video block is: the playback of the i-1th high-resolution video block is completed, and the ith high-resolution video block has been cached in the playback buffer;

[0100] Calculate the playback start time of the i-th high-resolution video block according to formula (4):

[0101]

[0102] Among them, t i represents the start time of playback of the i-th high-resolution video block; t i-1 represents the start time of playback of the i-1th high-resolution video block; T represents the time length of the high-resolution video block or the low-resolution video block; represents the cache start time of the i-th high-resolution video block cached to the playback buffer; d i represents the download time of the i-th high-resolution video chunk.

[0103] In a specific embodiment, when playing the high-resolution video blocks, if the high-resolution video blocks are exhausted in the playback buffer 110 , the high-resolution video blocks need to be re-cached;

[0104] If the high-resolution video block needs to be re-cached, the time for re-caching the high-resolution video block is calculated according to formula (5):

[0105] τ i =t i -(t i-1 +T) Formula (5)

[0106] Among them, τ i represents the time to re-cache the i-th high-resolution video chunk; t i represents the start time of playback of the i-th high-resolution video block; t i-1 represents the playback start time of the i-1th high-resolution video block; T represents the time length of the high-resolution video block or the low-resolution video block.

[0107] In a specific embodiment, the states of the low-resolution video block buffer, the high-resolution video block buffer, and the playback buffer at time t are shown in formulas (7) and (8):

[0108]

[0109]

[0110] Among them, B S (t) represents the status of the low-resolution video block buffer and the high-resolution video block buffer at time t; B p (t) represents the state of the playback buffer at time t; represents the reconstruction start time of the kth low-resolution video block; φ k (r k,r' k ) represents the reconstruction processing time for reconstructing the k-th low-resolution video block into a high-resolution video block; Indicates the cache start time of the k-th high-resolution video block in the playback buffer.

[0111] In a specific embodiment, after the high-resolution video block buffer and the playback buffer meet the conditions, the smoothness of the video streaming service is guaranteed, thereby avoiding video interruption. Under the premise that the smoothness of the video streaming service is guaranteed, in order to adapt to dynamic network conditions and the time-varying computing capabilities of edge nodes, the DRL agent needs to make decisions on the reconstruction video resolution level and the configuration of the SVSR model (including the model level and output path) at the same time. Therefore, the optimization goal is to maximize the user experience quality (QoE) of the video viewer. Since QoE is affected by three aspects: video quality, rebuffering time, and quality jitter between video blocks, these three quantifiable indicators can be used to define QoE. The specific steps are as follows:

[0112] (1) Setting the objective function: Build an optimization model that improves user video quality, reduces rebuffering time (rebuffering occurs when there are no unplayed video blocks in the playback buffer), and reduces quality jitter (video quality jitter is a key indicator of QoE because it can cause physiological symptoms such as dizziness and headaches in video viewers). The decision variables are the video reconstruction resolution level and the SVSR model configuration. Specifically, the objective function includes the following three parts:

[0113] Q1: Video quality can be defined as:

[0114]

[0115] Among them, B i ′ represents the bit rate of the i-th high-resolution video block after reconstruction. The dynamic adaptive streaming (DASH) system will correspond the resolution to the corresponding bit rate. F(B i ') indicates that the bit rate B i Mapped to the quality perceived by the user, N c Indicates the total number of video blocks received by the client during the entire system operation;

[0116] Q2: The rebuffering time can be defined as:

[0117]

[0118] Among them, τ i represents the time to re-cache the i-th high-resolution video block; N r Indicates the total number of rebuffering times during the entire system operation;

[0119] Q3: The quality jitter between two adjacent video blocks can be defined as:

[0120]

[0121] Among them, B′ i represents the bit rate of the i-th high-resolution video block after reconstruction; B′ i-1 N represents the bit rate of the i-1th high-resolution video block after reconstruction; c Indicates the total number of video blocks received by the client during the entire system operation.

[0122] When the video is heavily buffered and jittery, a penalty value will be applied. The longer the delay, the greater the penalty value. The quality of experience (QoE) for a single user can be expressed as:

[0123] QoE=Q1-μQ2-λQ3 (Formula 12)

[0124] Among them, μ and λ are weight factors.

[0125] In order to further prevent the terminal from interrupting playback, an underflow probability constraint can be added to the buffer length of the playback buffer, that is,

[0126]

[0127] in, Indicates the buffer length of the playback buffer when starting to rebuild the i-th video block, B bound The minimum threshold of the buffer length of the playback buffer area is B. bound When , there is a possibility of video interruption, and ε2 is the probability of constraint violation.

[0128] In summary, the optimization problem can be expressed as:

[0129] max E{QoE}=E{Q1-μQ2-λQ3}

[0130] stP(μ δ <μ′ δ )≤ε1

[0131]

[0132] Where Q1 is the video quality; Q2 is the rebuffering time; Q3 is the quality jitter caused by the resolution switching of adjacent video blocks; μ and λ are weight factors; μ′ δ is the computing power required by the edge node to reconstruct the video block; ε1 is the probability constraint threshold of the constraint; QoE is the quality of experience of a single user; B bound is the minimum threshold of the video playback duration in the playback cache queue; ε2 is the probability of constraint violation.

[0133] (2) Reinforcement learning is used to solve the optimization problem. The intelligent edge nodes supporting reinforcement learning are trained until the algorithm converges. The trained reinforcement learning algorithm is used to make decisions on the resolution level of the reconstructed video and the configuration of the SVSR model. The specific steps are as follows:

[0134] The high-resolution video block playback process is a Markov decision problem. Reinforcement learning does not require any pre-defined data. Instead, it obtains learning information and updates model parameters by receiving rewards (feedback) from the environment for actions. Therefore, reinforcement learning can be used to solve optimization problems.

[0135] ② State space. The state of SVSR when it starts to reconstruct the i-th video block can be expressed as

[0136]

[0137] in, and They represent the length of the playback buffer and the high-resolution video block buffer when starting to reconstruct the i-th video block; the main limitation of the decision is bandwidth, is the average download bandwidth of the past k1 video chunks; is the total waiting time from when k2 video chunks were requested to when they were downloaded; is the implicit available computing power of the past k3 video blocks; r′ i-1 is the resolution of the i-1th video block; Ni is the number of remaining blocks in the video block.

[0138] ③ Action space. For each video block, the DRL agent must select the target video resolution for reconstruction based on the current state and the most appropriate SVSR model based on the currently available computing resources. The size of the VSR model and the target video resolution will affect the consumption of computing resources on the edge node. Therefore, the DRL agent in SVSR needs to make decisions including 1) the video reconstruction resolution level (r i ,r′ i ), 2) Configuration of SVSR model (including model level m i and output path l i ). The action vector of SVSR when it starts to reconstruct the i-th video block can be expressed as

[0139] ④ Reward function. The DRL agent makes adaptive decisions based on the collected real-time system status. After the system performs an action, an immediate reward is fed back to the DRL agent. In order to enable users to obtain a better video service experience and achieve the goal of maximizing user QoE, the reward function is defined as

[0140] Ri =B′ i -μτ i -λ|B′ i -B′ i-1 |

[0141] Among them, B′ i represents the resolution of the i-th video block after reconstruction; τ i represents the rebuffering time that occurs when processing the i-th video chunk; |B′ i -B′ i-1 | represents the quality jitter between the i-th video block and the i-1-th video block; μ and λ are weight factors.

[0142] ⑤Deep neural network model and model training.

[0143] The Bellman equation for deep reinforcement learning can be written as

[0144]

[0145] For Markov decision processes (MDPs) with a large number of states and actions, deep neural networks Q(s i ,a i ;θ) to approximate Q(s i ,a i ), θ is a parameter.

[0146] For the deep neural network Q(s i ,a i ; θ) is trained according to the DQN (Deep Q-Network) algorithm.

[0147] DQN contains two neural networks: an evaluation network (eval_net) and a target network (target_net). The evaluation network is used to evaluate each action a in the current state. i The Q value of the network is used to select action a i , select the action a with the largest Q value i After that, the environment feeds back the reward and the next state s i+1 , that is (s i ,a i ,R i ,s i+1 ) and then store it. Input s to the target network i+1 , that is, the next environmental state, so as to obtain a under each action through the network i The Q value of the state s i Next action a i The reward R i , and the Bellman equation to calculate the target, we get Q(si ,a i ).

[0148] The DQN algorithm combines neural networks with Q-learning and uses neural networks to approximate the function Q(s i ,a i ), the input is state s i , the output is for each action a i The corresponding Q value is then used to select the action to be performed in the corresponding state according to the size of the Q value to complete the control.

[0149] Status i Input, get the Q value Q(s) corresponding to all actions i ,a i ); select the action a′ with the largest Q value i And execute; after execution, the environment changes and the reward R of the environment can be obtained i ;Use reward R i Update Q(s i ,a′ i ), using the new Q(s i ,a′ i )Update network parameters.

[0150] ⑥ After the model is trained, it can collect real-time state information as input and output the optimal control strategy, including 1) video reconstruction resolution level (r i ,r′ i ), 2) Configuration of SVSR model (including model level m i and output path l i ).

[0151] Example 2

[0152] like Figure 3 As shown, this embodiment 2 provides a video transmission system for a scalable video super-resolution model. The video transmission system is used to implement the above-mentioned video transmission method based on the scalable video super-resolution model. The video transmission system includes:

[0153] The client 100 is configured to send a request signal and, when first connecting to the edge node 200, to send the device type and maximum display resolution along with the request signal to the edge node 200. The client 100 needs to periodically report the playback status to the edge node 200. The client 100 is also configured to receive and play high-resolution video blocks.

[0154] The server 300 is configured to receive a request signal and send a low-resolution video block.

[0155] edge node 200,

[0156] The edge node 200 is in communication with the client 100 and is configured to receive a request signal sent by the client 100 and send the request signal to the server 300;

[0157] The edge node 200 is in communication with the server 300 , and is configured to receive low-resolution video blocks sent by the server 300 , reconstruct the low-resolution video blocks to obtain high-resolution video blocks, and send the high-resolution video blocks to the client 100 .

[0158] The above-mentioned edge node 200 can collect network status information, has the ability to support scalable video super-resolution processing, and can adaptively adjust the model size and reconstruction resolution of the scalable video super-resolution model, integrating edge computing and video super-resolution to improve video quality.

[0159] In a specific embodiment, the client 100 sends a request signal to the edge node 200, and the edge node 200 sends the request signal of the client 100 to the server 300. After receiving the request signal, the server 300 provides the edge node 200 with the lowest resolution video block of the request signal. Then, the edge node 200 receives the lowest resolution video block, determines the model size and reconstruction resolution of the scalable video super-resolution model through DRL processing, and the SVSR processor 230 uses the determined scalable video super-resolution model to reconstruct the lowest resolution video block to obtain a high-resolution video block, caches the high-resolution video block to the high-resolution video block buffer 233, and finally sends the high-resolution video block to the client 100, thereby completing video playback.

[0160] In a specific embodiment, the server 300 is used to store low-resolution videos. According to the DASH standard, the low-resolution videos are divided into N low-resolution video blocks with the same time length T. The N low-resolution video blocks are encoded into a group of resolutions to obtain a low-resolution video block group consisting of N low-resolution video blocks.

[0161] In a specific embodiment, the edge node 200 includes a monitor 210, a DRL agent 220, and an SVSR processor 230;

[0162] The monitor 210 is in communication with the client 100 and the DRL agent 220 , and is used to monitor the network status of the server 300 , the edge node 200 and the client 100 ;

[0163] The DRL agent 220 is in communication with the server 300, and the DRL agent 220 is used to receive network status information and determine the model size and reconstruction resolution of the scalable video super-resolution model according to the network status information;

[0164] The above-mentioned SVSR processor 230 is communicatively connected with the DRL agent 220, the server 300 and the client 100. The above-mentioned SVSR processor 230 can support the SVSR algorithm based on deep learning, and is used to reconstruct the low-resolution video block using the above-mentioned scalable video super-resolution model after determining the model size and reconstruction resolution to obtain a high-resolution video block.

[0165] In a specific embodiment, the SVSR processor 230 includes a low-resolution video block buffer 231 , a high-resolution video block buffer 233 , and a processing area 232 ;

[0166] The low-resolution video block buffer 231 is used to cache the low-resolution video blocks sent by the server 300 to the SVSR processor 230;

[0167] The low-resolution video blocks in the low-resolution video block buffer 231 are sequentially reconstructed in the processing area 232 according to the first-in-first-out principle to obtain high-resolution video blocks;

[0168] The high-resolution video block is cached in the high-resolution video block buffer 233 .

[0169] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video transmission method based on a scalable video super-resolution model, characterized in that: The following steps are involved: Collecting network status information, and determining a model size and a reconstruction resolution of a scalable video super-resolution model based on the network status information and available computing resources on the edge node; After receiving the request signal, the server feeds back the corresponding low-resolution video block according to the request signal; The server transmits the corresponding low-resolution video blocks to the edge nodes; Reconstructing low-resolution video blocks on edge nodes using the scalable video super-resolution model after determining the model size and reconstruction resolution to obtain high-resolution video blocks; After transferring the high-resolution video block to a playback buffer, playing the high-resolution video block; The specific steps of determining the model size and reconstruction resolution of the scalable video super-resolution model according to the network status information are as follows: Pre-configure a group of neural network models, and select a matching neural network model from the group of neural network models according to the number of residual convolution blocks; Determining an output path for exiting a scalable video super-resolution model based on output features of different numbers of residual convolutional blocks; The low-resolution video blocks are reconstructed using the scalable video super-resolution model after determining the model size and reconstruction resolution. The specific steps are as follows: caching the low-resolution video blocks in a low-resolution video block buffer, and sequentially reconstructing the low-resolution video blocks in the low-resolution video block buffer according to a first-in-first-out principle to obtain high-resolution video blocks; caching the high-resolution video block in a high-resolution video block buffer; Calculate the reconstruction start time of the i-th low-resolution video block.

2. The video transmission method based on the scalable video super-resolution model according to claim 1, characterized in that The cache start time for caching the high-resolution video block in the playback buffer is calculated as follows: after the i-th low-resolution video block is reconstructed and the (i-1)-th high-resolution video block is transferred to the playback buffer, the i-th high-resolution video block begins to be transferred to the playback buffer.

3. The video transmission method based on the scalable video super-resolution model according to claim 2, characterized in that Playback time of the high-resolution video block: when the (i-1)th high-resolution video block is played and the i-th high-resolution video block has been cached in the playback buffer, the playback start time of the i-th high-resolution video block is calculated.

4. The video transmission method based on the scalable video super-resolution model according to claim 3, characterized in that When playing the high-resolution video block, if the high-resolution video block is exhausted in the playback buffer, the high-resolution video block needs to be re-cached. If the high-resolution video block needs to be re-cached, the time for re-caching the high-resolution video block is calculated.

5. A video transmission system for a scalable video super-resolution model, characterized in that: The video transmission system is used to implement the video transmission method based on a scalable video super-resolution model according to any one of claims 1 to 4, and the video transmission system includes: A client (100) for sending request signals and receiving high-resolution video chunks; A server (300) for receiving a request signal and sending a low-resolution video block; Edge Node (200), The edge node (200) is in communication with the client (100), and is configured to receive a request signal sent by the client (100), and send the request signal to the server (300); The edge node (200) is in communication with the server (300) and is configured to receive low-resolution video blocks sent by the server (300), reconstruct the low-resolution video blocks to obtain high-resolution video blocks, and send the high-resolution video blocks to the client (100).

6. The video transmission system of the scalable video super-resolution model according to claim 5, characterized in that The server (300) is used for storing low-resolution videos, dividing the low-resolution videos into N low-resolution video blocks with the same time length T, and obtaining a low-resolution video block group consisting of the N low-resolution video blocks.

7. The video transmission system of the scalable video super-resolution model according to claim 5, characterized in that The edge node (200) includes a monitor (210), a DRL agent (220) and an SVSR processor (230); The monitor (210) is in communication with the client (100), the DRL agent (220) and the server (300), and the monitor (210) is used to monitor the network status of the server (300), the edge node (200) and the client (100); The DRL agent (220) is used to receive network status information and determine the model size and reconstruction resolution of the scalable video super-resolution model according to the network status information; The SVSR processor (230) is in communication with the DRL agent (220), and the SVSR processor (230) is used to reconstruct low-resolution video blocks using the scalable video super-resolution model after determining the model size and reconstruction resolution to obtain high-resolution video blocks.

8. The video transmission system of the scalable video super-resolution model according to claim 7, characterized in that The SVSR processor (230) includes a low-resolution video block buffer (231), a high-resolution video block buffer (233) and a processing area (232); The low-resolution video block buffer (231) is used to cache the low-resolution video blocks sent by the server (300) to the SVSR processor (230); Reconstructing the low-resolution video blocks in the low-resolution video block buffer (231) in sequence according to a first-in-first-out principle in the processing area (232) to obtain high-resolution video blocks; The high-resolution video blocks are cached in the high-resolution video block buffer (233).

Images