Self-adaptive super-resolution method, device and system for mobile terminal video

By deploying adaptive super-resolution models and reinforcement learning decisions on mobile devices, dynamically adjusting the bit rate and multiple of video blocks, the problem of video quality and smoothness balance in mobile device video transmission is solved, and user experience and device performance is improved.

CN119094844APending Publication Date: 2024-12-06NANJING UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411119760.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

When mobile devices transmit video in low bandwidth environments, the prior art is difficult to balance maintaining video quality and smoothness, resulting in limited user experience and high energy overhead and inference delays of super-resolution models.

Method used

Adaptive super-resolution method for mobile terminals is adopted to dynamically adjust the bit rate and super-segment multiple of video blocks by establishing multiple magnification super-resolution models and reinforcement learning intelligent decision-making bodies, combining shallow and deep feature extraction, reconstruction and post- Processing, optimize video transmission performance.

Benefits of technology

It improves the performance of mobile device video transmission and user viewing experience, balances video quality, fluency and energy consumption, and adapts to different bandwidths and device environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094844A_ABST
    Figure CN119094844A_ABST
Patent Text Reader

Abstract

The invention discloses a self-adaptive super-resolution method and device for a mobile terminal video. The method comprises the following steps: establishing a mathematical model for a mobile terminal video transmission scene based on super-resolution and a dynamic bit rate; establishing a super-resolution model suitable for a mobile terminal; establishing a reinforcement learning agent model for executing super-division multiple and bit rate decisions; building a system for pre-training the super-resolution model and the intelligent agent by using a large data set; and designing an adaptive retraining mechanism and method, and issuing decisions of the super-resolution model and the intelligent agent to mobile terminal equipment for video pulling and watching. According to the method, the energy consumption and the video transmission effect of the mobile terminal are comprehensively considered, the computing power of equipment is utilized under the condition that the bandwidth fluctuation of the mobile terminal is large, the video quality and the loading fluency are enhanced, and the overall video watching experience of a user is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an adaptive super-resolution method, device and system for mobile terminal videos. Background Art

[0002] As the number of mobile users continues to increase, the demand for mobile video streaming has grown exponentially, with mobile devices contributing more than 75% of video viewing. However, the ever-changing network environment poses a major challenge to smooth video playback for mobile users. The instability and fluctuation of mobile networks often lead to problems such as video freezes, blurs, and extended loading times, which seriously affect user satisfaction and engagement. Mobile users have higher and higher expectations for video quality, prompting video streaming service providers to strive for higher resolution and better user experience.

[0003] Dynamic Adaptive Streaming over HTTP (DASH) is a widely used video streaming technology that uses adaptive bitrate streaming (ABR) algorithms to improve user experience (Quality of Experience, QoE) under fluctuating network conditions. These ABR algorithms collectively aim to determine the optimal bitrate for each video chunk, taking into account the current network conditions. However, they face challenges in maintaining QoE in situations where bandwidth is limited, as they must strike a delicate balance between video smoothness and quality.

[0004] Super-resolution (SR) is a powerful tool that can reconstruct high-resolution images from low-resolution images. Researchers have investigated the possibility of applying SR models to videos, and recent research has focused on leveraging these models to enhance video streaming. Based on SR models, we have the ability to upsample low-resolution video blocks acquired in low-bandwidth environments. However, this process introduces additional energy overhead and may increase inference latency, thereby reducing user QoE. Therefore, it is crucial to optimize the SR model for mobile devices before performing client-side upsampling. The goal is to minimize energy overhead and inference time while maintaining improved video quality. In addition, the clarity of the acquired video blocks also affects the complexity of running SR. Determining the bitrate of the video blocks to be acquired and the upsampling factor to be applied becomes a challenging task. There is a need in the prior art for a video transmission scheme that balances energy consumption and video quality. Summary of the invention

[0005] Purpose of the invention: In view of the shortcomings of the prior art, the present invention proposes an adaptive super-resolution method, device and system for mobile video, which performs dynamic resolution selection and super-resolution of a specified ratio on the client during video transmission, effectively improving the mobile video transmission performance under different bandwidth conditions.

[0006] Technical solution: According to a first aspect of the present invention, an adaptive super-resolution method for mobile terminal videos is provided, comprising the following steps:

[0007] Establish a multi-rate super-resolution model adapted to mobile devices, use a large-scale dataset for pre-training, and extract specified frames from each target video to retrain the model. The trained super-resolution model is deployed on mobile devices.

[0008] A reinforcement learning intelligent decision-making body model is established to perform super-resolution multiples and bit rate decisions. The behavior a of the intelligent decision-making body contains the bit rate b of the video block to be downloaded and the super-resolution multiple u. The environmental state information s obtained by the intelligent decision-making body before making a decision contains the size and video quality of different bit rates of the next data block, the buffer size, and the time and energy consumption of loading the previous data block. The goal is to maximize the cumulative reward R of each action i , the R i An intelligent decision-making agent model that is associated with the user's video viewing experience QoE and trained based on real data sets. The decisions made by the agent are transmitted to mobile devices.

[0009] The video streaming data of each user is continuously collected, including instant rewards, rebuffering time and estimated bandwidth. When the system performance degrades and meets the specified conditions, the intelligent decision-maker uses the historical video blocks of the current user to update its policy network and uses the new policy network to select the next action, so that the video quality can adapt to different mobile devices and networks.

[0010] Furthermore, the super-resolution model consists of four main parts: shallow feature extraction, deep feature extraction, reconstruction and post-processing. The shallow feature extraction part uses a convolutional layer and a PReLU activation function to transform the input low-resolution image I L Convert it into feature space to obtain the first feature map F0; the deep feature extraction part uses two pairs of convolutional layers and PReLU activation function layers to extract the second feature map F0 from the first feature map i ; The reconstruction part uses a convolutional layer as a transition layer to transform the second feature map feature F i Map back to the high-resolution image space to obtain the residual image feature F t , and the residual image feature F t Add it to the input low-resolution image to get the super-resolution image F S; The post-processing part rearranges the image through a pixel rearrangement layer and clamps the values ​​to a normal range through a clip node.

[0011] Furthermore, the loss function in the super-resolution model training process adopts the L1 loss function:

[0012]

[0013] Where θ represents the parameters of the current super-resolution model, N represents the number of video frames used for training, represents the super-resolution result of the i-th frame, Represents the original high-resolution image of the i-th frame.

[0014] Furthermore, in the intelligent decision-making model, the cumulative reward R of the action i By instant reward QoE for all possible future video chunks k k The weighted sum is calculated using the discount factor γ:

[0015]

[0016] Among them, QoE k represents the viewing experience of the user of the current k-th video block, and its calculation method is:

[0017] QoE k =α×q k -β×(q k -q k-1 )-η×p k -ω×E k

[0018] In the formula, q k represents the video quality of the kth video block, p k represents the penalty for the kth video freeze, E i represents the energy consumption of playing the kth video block, and α, β, η, and ω represent the impact coefficients of video quality, smoothness, jamming, and energy consumption on QoE, respectively.

[0019] Furthermore, advantage estimation is used to guide strategy updates during the training of the intelligent decision-making model to improve the decision-making ability of the intelligent agent, including:

[0020] The general advantage estimation GAE method is adopted, and the λ hyperparameter is introduced to balance the bias and variance. The advantage function The definition of is based on the immediate reward, value function, and the advantage of the next state action:

[0021]

[0022] Among them, QoE irepresents the instant reward, which is the viewing experience of the user of the kth video block, V φ (s) represents the total reward that can be obtained from state s under the current strategy φ, λ is the GAE parameter, s ′ and a ′ They represent the next state and action in the current video transmission process respectively;

[0023] Combining the strategy loss and value loss of PPO, the strategy loss measures the product of the probability ratio of the current strategy network and the old strategy network to select actions and the advantage estimate. The strategy loss is calculated by the following formula:

[0024]

[0025] where L(s,a,θ ′ ,θ) is the current state s, the selected action a, and the current policy network parameters θ and the old policy network parameters θ ′ The loss function, π θ (a, s) refers to the probability of selecting action a in state s according to the current policy network θ, and (s,a) is the old policy network θ ′ The advantage estimate of action a in state s, is a clipped version of the advantage function, where:

[0026]

[0027] Where ∈ is a positive value introduced to balance the policy update performance.

[0028] Furthermore, the adaptive retraining includes: using a counter to record the number of times that the instant rewards of the mobile terminal device for multiple consecutive data blocks are negative, when the counter reaches a threshold δ, activating an online retraining process, and after the retraining process, resetting the counter to 0.

[0029] According to a second aspect of the present invention, there is provided an adaptive super-resolution service device for mobile terminal videos, comprising:

[0030] The super-resolution model building and training module is configured to build a multi-rate super-resolution model adapted to the mobile terminal, use a large-scale data set for pre-training, and extract a specified frame for each target video to re-train the model. The trained super-resolution model is used to be deployed on the mobile terminal device;

[0031] The intelligent decision-making agent model construction and training module is configured to build a reinforcement learning agent model that performs super-resolution multiples and bit rate decisions. The behavior a of the intelligent decision-making agent includes the bit rate b of the video block to be downloaded and the super-resolution multiple u. The environmental state information s obtained by the intelligent decision-making agent before making a decision includes the size of the next data block at different bit rates and video quality, the buffer size, and the time and energy consumption of loading the previous data block. The goal is to maximize the cumulative reward R for each action. i , the R i An intelligent decision-making agent model that is associated with the user's video viewing experience QoE and trained based on real data sets. The decisions made by the agent are transmitted to mobile devices.

[0032] The adaptive retraining module is configured to continuously collect video stream data for each user, including immediate rewards, rebuffering time, and estimated bandwidth. When the system performance degrades and meets the specified conditions, the intelligent decision-maker uses the historical video blocks of the current user to update its policy network and uses the new policy network to select the next action, so that the video quality can adapt to different mobile devices and networks.

[0033] According to a third aspect of the present invention, there is provided an intelligent decision-making and super-resolution service device, the device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the adaptive super-resolution method for mobile video described in the first aspect are implemented.

[0034] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the adaptive super-resolution method for mobile terminal video as described in the first aspect of the present invention are implemented.

[0035] According to a fifth aspect of the present invention, there is provided a video transmission system, comprising a content distribution network, a server, and a client, wherein the server is configured to execute the adaptive super-resolution method for mobile terminal video as described in the first aspect, or to adopt the super-resolution service device as described in the second aspect, or to adopt the intelligent decision-making and super-resolution service device as described in the third aspect, and to send the super-resolution model and the decision made by the intelligent agent to the client, wherein the client is a mobile terminal device on which a super-resolution model trained by the server is deployed, and the mobile terminal device takes corresponding actions according to the decision of the intelligent decision-making body of the server, and pulls videos from the content distribution network for users to watch.

[0036] Beneficial effects: To address the problem of stuttering in mobile video transmission, the present invention combines adaptive bitrate and super-resolution models to enhance the user's viewing experience. By establishing a suitable system model and setting optimization goals, the bitrate and super-resolution selection are dynamically adjusted based on a reinforcement learning algorithm. A lightweight super-resolution model is built to ensure its lightweight characteristics so that super-resolution can be efficiently performed on mobile devices with limited resources. The present invention fills a gap in the field, while taking into account video quality, smoothness, and overall energy consumption, and selecting a suitable bitrate and super-resolution. The present invention has a wide range of applicability and can effectively improve the video transmission effect of mobile devices and the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the adaptive super-resolution method for mobile terminal video of the present invention;

[0038] Figure 2 is a schematic diagram of a mobile terminal super-resolution model of the present invention;

[0039] Figure 3 It is a schematic diagram of the super-resolution multiple and bit rate decision-making intelligent agent based on reinforcement learning of the present invention. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0041] Reference Figure 1 The present invention proposes an adaptive super-resolution system for mobile video, including an intelligent decision-making body deployed on the server side after training with an existing data set, and a client deployed on the mobile device that can super-resolution the video segment. It combines the super-resolution processing capability of the client and the precise control of the super-resolution ratio and bit rate by the server-side decision-making system. Similar to the traditional dynamic bit rate video transmission system, the system uses 4-second segments as the basic transmission unit to ensure that the adaptive super-resolution operation of the video can be refined.

[0042] In the early stage of video distribution, the server will pre-process video clips of different bit rates and store them efficiently in the content delivery network (CDN) for clients to download on demand. In addition, the server uses the low bit rate and high bit rate versions of the video to train the super-resolution model, ensuring that the advanced super-resolution model has been transmitted and deployed to the client before the video is played.

[0043] Before the client starts playing the video, it first loads the super-resolution model and, after receiving the super-resolution ratio instruction calculated by the server, starts super-resolution processing for the downloaded low-resolution video clips, substantially improving the video clarity and greatly improving the user's viewing experience. In general, this system achieves the enhancement of video quality in each link of the video transmission chain through intelligent and adaptive methods.

[0044] In order to help the client make the correct bit rate and super-resolution ratio selection, the server uses a large real data set to pre-train a general intelligent agent, which makes appropriate decisions by inputting the user's video transmission environment status. The server will transmit the decision made by the intelligent agent to the client, so that the client can pull the video segment of the specified bit rate from the CDN and perform super-resolution at an appropriate ratio. At the same time, in order to take into account the differences in device performance and network conditions between different users, the server will retrain the intelligent agent at an appropriate time to ensure that the correct decision is made to improve the user's viewing experience.

[0045] Reference Figure 2 The present invention proposes a mobile terminal super-resolution model, which is executed on a user terminal with limited computing resources and mainly includes the following steps:

[0046] S1. Build a super-resolution neural network for mobile terminals: It includes four main parts: shallow feature extraction, deep feature extraction, reconstruction and post-processing. First, through the shallow feature extraction part, the input image is converted into feature space. This step is completed using a 3×3 convolution layer and a PReLU activation function. L (Input Low Resolution) converted to feature space:

[0047] F0=f l (I L )

[0048] Among them, F0 is the temporary output result of the first layer, I L is a low-resolution representation of the image, f l It represents the mapping function involved in shallow feature extraction, including a 3×3 convolution and PReLU. PReLU is a commonly used activation function, which can be expressed as PReLU(x)=max(0,x)+αmin(0,x).

[0049] Furthermore, the deep feature extraction part is used to gradually extract high-level information and restore details (such as edges and textures). The deep feature extraction part contains two pairs of 3×3 convolution and PReLU activation function layers to fully utilize parallel reasoning and match the overhead of the upper layer:

[0050] F i =fh (F0),i=1,2

[0051] where F i represents the intermediate value after the i-th pair of convolution and activation, f h It is the mapping function of the deep feature extraction part.

[0052] Furthermore, after feature extraction is completed, a 3×3 convolutional layer is used again as a transition layer to map the features back to the high-resolution image space to obtain the residual image features:

[0053] F t =f t (F2)

[0054] where f t represents the mapping function in the transition layer, F t is the obtained residual image feature.

[0055] Furthermore, these residual features are added to the input low-resolution image to obtain a super-resolution image:

[0056] F S =F t +I L

[0057] Finally, the image is rearranged by a pixel rearrangement layer and the values ​​are clamped within a normal range by a clip node to ensure the quality of the output image.

[0058] The loss function during model training uses the L1 loss function to optimize the network:

[0059]

[0060] Where θ represents the parameters of the current super-resolution model, N represents the number of video frames used for training, represents the super-resolution result of the i-th frame, Represents the original high-resolution image of the i-th frame. This loss function measures the difference between the network output and the true high-resolution image, which helps the network learn to generate more accurate super-resolution images. This designed architecture can effectively improve the quality and authenticity of super-resolution images, thereby enhancing the user's visual experience.

[0061] S2. Use a large-scale dataset to pre-train the super-resolution model to obtain a benchmark model:

[0062] S2-1. Data preprocessing: Select a broad and representative dataset, such as DIV2K, which contains high-resolution images of different scenes and objects, suitable for pre-training of super-resolution tasks. DIV2K is a commonly used dataset containing 2,000 2K resolution images collected on various devices. Perform various preprocessing operations on the images in the dataset, including scaling, cropping, and normalization, to adapt to the input requirements during training.

[0063] S2-2, Model training: Use the selected dataset to train the model, which includes forward propagation, calculating the loss function, and backpropagation to update the weights. The training process may include multiple rounds of iterations until the model converges.

[0064] S2-3. Model validation: Evaluate the performance of the model on an independent validation set to ensure that the model not only performs well on the training set but also has good generalization capabilities.

[0065] S3. Retrain the model using the high bitrate version and the low bitrate version of the target video:

[0066] S3-1, target video selection and preprocessing: select the target video that needs to be super-resolution processed, extract image frames from it, and then perform the same preprocessing steps as before on these images.

[0067] S3-2. Load pre-trained model: Load a pre-trained model that has been trained on a large-scale dataset, and load the previously saved weights and structure.

[0068] S3-3. Model fine-tuning: Fine-tune the model using pre-processed images of the target video, using a lower learning rate and a shorter training time, in order to adapt the model to the special properties of the specific content rather than learning from scratch. Use other image frames of the target video to evaluate the performance of the fine-tuned model to ensure that it can effectively improve the resolution of the video.

[0069] Reference Figure 3 The super-resolution multiple and bit rate decision agent based on reinforcement learning proposed in the present invention is executed on the server side and mainly includes the following steps:

[0070] S4. Based on the PPO (Proximal Policy Optimization) reinforcement learning algorithm model, an intelligent decision-making body for dynamic bit rate and super-resolution ratio is established to perform specific modeling of QoE.

[0071] S4-1. In order to ensure that the agent can make correct decisions, the QoE calculation formula is defined as follows:

[0072] QoE k =α×q k-β×(q k -q k-1 )-η×p k -ω×E k

[0073] Among them, QoE k represents the viewing experience of the user of the current k-th video block, q k represents the video quality of the kth video block, which can be evaluated by VMAF (Video Muitimethod Assessment Fusion), p k Indicates the penalty for jamming (rebuffering), E i represents the energy consumption of playing the kth video block. α, β, η, ω represent the impact coefficients of video quality, smoothness, stuttering, and energy consumption on QoE, respectively. In order to balance the impact of energy consumption and viewing experience, they can be set to 0.04, 0.01, 0.2, and 0.0005.

[0074] S4-2. It is necessary to set a suitable training environment and objective function for the intelligent decision-making body to help it learn to make appropriate decisions under specific conditions, that is, to decide on the optimal resolution and super-resolution ratio under different environments.

[0075] The final behavior a of the intelligent decision-making body contains two dimensions: the bit rate b of the video block to be downloaded and the super-resolution multiple u. The environmental state information s that the intelligent agent can capture before making a decision includes: the size and video quality of the different bit rates of the next data block, the buffer size, and the time and energy consumption of loading the previous data block.

[0076] The goal is to maximize the cumulative reward R for each action i , which is calculated by the immediate reward QoE for all possible future video chunks k k It is calculated by weighting and summing the discount factor γ, that is:

[0077]

[0078] The present invention uses advantage estimation to guide strategy updating and improve the decision-making ability of the intelligent agent.

[0079] Specifically, the general advantage estimation (GAE) method proposed by Schlomann et al. was adopted. By introducing the λ hyperparameter to balance the bias and variance, the advantage function The definition of is based on the immediate reward, value function, and the advantage of the next state action:

[0080]

[0081] Among them, QoEk represents the immediate reward obtained by the agent for block k, V φ (s) represents the total reward that can be obtained from state s under the current strategy φ, λ is the GAE parameter, s ′ and a ′ Represent the next state and action in the current video transmission process, respectively. By incorporating the GAE parameter λ into the advantage estimate, we can control the trade-off between bias and variance and fine-tune the advantage estimate according to the specific requirements of our agent training.

[0082] S4-3, the actor network is responsible for outputting the distribution of an action according to the current environment state s, that is, the probability of selecting each bit rate and super-resolution multiple combination. The critic network is responsible for evaluating the value of the current strategy in a given state, that is, predicting the cumulative return that can be obtained by taking the current strategy starting from the current state s.

[0083] Combining the strategy loss and value loss of PPO, the strategy loss measures the product of the probability ratio of the current strategy network and the old strategy network to select actions and the advantage estimate. The strategy loss is calculated by the following formula:

[0084]

[0085] where L(s,a,θ ′ ,θ) is the current state s, the selected action a, and the current policy network parameters θ and the old policy network parameters θ ′ The loss function of π θ (a, s) refers to the probability of selecting action a in state s according to the current policy network θ, and is based on the old policy network θ ′ Advantage estimate of action a in state s. is a clipped version of the advantage function, where:

[0086]

[0087] To ensure that the policy update is balanced, neither too conservative nor too aggressive, it is common practice to introduce a small positive value, denoted as ∈, during the clipping process. This value is usually set to 0.1 and plays a crucial role in controlling the magnitude of the policy update.

[0088] S5. Adaptive retraining is introduced to adapt to different mobile devices and networks. By continuously collecting video streaming data for each user, including instant rewards, rebuffering time, and estimated bandwidth (throughput). When the system performance degrades significantly, an adaptive online retraining process is initiated. Specifically, a counter is set on the client to record the number of times the instant rewards of several consecutive data blocks are negative. When the counter reaches the threshold δ, which is set to 3 in the experiment, the online retraining process is activated. During online retraining, the intelligent decision-maker on the server uses the real-world network trajectories collected from the past 100 video blocks of the current user to update its policy network. The new policy network is then used to select the next action. After the retraining process, the counter is reset to 0.

[0089] Based on the same technical concept as the method embodiment, the present invention also provides an adaptive super-resolution service device for mobile terminal videos, including:

[0090] The super-resolution model training module is configured to establish a multi-magnification super-resolution model adapted to the mobile terminal, use a large-scale data set for pre-training, and extract a specified frame for each target video to re-train the model. The trained super-resolution model is used to be deployed on the mobile terminal device;

[0091] The intelligent decision-making agent model training module is configured to establish a reinforcement learning agent model that performs super-resolution multiples and bit rate decisions. The behavior a of the intelligent decision-making agent includes the bit rate b of the video block to be downloaded and the super-resolution multiple u. The environmental state information s obtained by the intelligent decision-making agent before making a decision includes the size of the next data block at different bit rates and video quality, the buffer size, and the time and energy consumption of loading the previous data block. The goal is to maximize the cumulative reward R for each action. i , the R i An intelligent decision-making agent model that is associated with the user's video viewing experience QoE and trained based on real data sets. The decisions made by the agent are transmitted to mobile devices.

[0092] The adaptive retraining module is configured to continuously collect video stream data for each user, including immediate rewards, rebuffering time, and estimated bandwidth. When the system performance degrades and meets the specified conditions, the intelligent decision-maker uses the historical video blocks of the current user to update its policy network and uses the new policy network to select the next action, so that the video quality can adapt to different mobile devices and networks.

[0093] The present invention also provides an intelligent decision-making and super-resolution service device, which includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the adaptive super-resolution method for mobile video described in the first aspect are implemented.

[0094] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the adaptive super-resolution method for mobile terminal video as described in the first aspect of the present invention are implemented.

[0095] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices (systems), computer equipment or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0096] The present invention is described with reference to a flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and a combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A device that specifies functions in a process or multiple processes.

[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 The steps of a specified function in a process or multiple processes.

Claims

1. An adaptive super-resolution method for mobile terminal videos, characterized in that: The following steps are involved: Establish a multi-rate super-resolution model adapted to mobile devices, use a large-scale dataset for pre-training, and extract specified frames from each target video to retrain the model. The trained super-resolution model is deployed on mobile devices. A reinforcement learning intelligent decision-making body model is established to perform super-resolution multiples and bit rate decisions. The behavior a of the intelligent decision-making body contains the bit rate b of the video block to be downloaded and the super-resolution multiple u. The environmental state information s obtained by the intelligent decision-making body before making a decision contains the size and video quality of different bit rates of the next data block, the buffer size, and the time and energy consumption of loading the previous data block. The goal is to maximize the cumulative reward R of each action i , the R i An intelligent decision-making agent model that is associated with the user's video viewing experience QoE and trained based on real data sets. The decisions made by the agent are transmitted to mobile devices. The video streaming data of each user is continuously collected, including instant rewards, rebuffering time and estimated bandwidth. When the system performance degrades and meets the specified conditions, the intelligent decision-maker uses the historical video blocks of the current user to update its policy network and uses the new policy network to select the next action, so that the video quality can adapt to different mobile devices and networks.

2. The method according to claim 1, characterized in that The super-resolution model consists of four main parts: shallow feature extraction, deep feature extraction, reconstruction and post-processing. The shallow feature extraction part uses a convolutional layer and PReLU activation function to transform the input low-resolution image I L Convert it into feature space to obtain the first feature map F0; the deep feature extraction part uses two pairs of convolutional layers and PReLU activation function layers to extract the second feature map F0 from the first feature map i ; The reconstruction part uses a convolutional layer as a transition layer to transform the second feature map feature F i Map back to the high-resolution image space to obtain the residual image feature F t , and the residual image feature F t Add it to the input low-resolution image to get the super-resolution image F S ; The post-processing part rearranges the image through a pixel rearrangement layer and clamps the values ​​to a normal range through a clip node.

3. The method according to claim 1, characterized in that The loss function used in the super-resolution model training process is the L1 loss function: Where θ represents the parameters of the current super-resolution model, N represents the number of video frames used for training, represents the super-resolution result of the i-th frame, Represents the original high-resolution image of the i-th frame.

4. The method according to claim 1, characterized in that In the intelligent decision-making model, the cumulative reward R of the action i By instant reward QoE for all possible future video chunks k k The weighted sum is calculated using the discount factor γ: Among them, QoE k represents the viewing experience of the user of the current k-th video block, and its calculation method is: Quality k =α×q k -β×(q k -q k-1 )-η×p k -ω×E k In the formula, q k represents the video quality of the kth video block, p k represents the penalty for the kth video freeze, E i represents the energy consumption of playing the kth video block, and α, β, η, and ω represent the impact coefficients of video quality, smoothness, jamming, and energy consumption on QoE, respectively.

5. The method according to claim 1, characterized in that During the training process of the intelligent decision-making model, advantage estimation is used to guide strategy updates and improve the decision-making ability of the intelligent agent, including: The general advantage estimation GAE method is adopted, and the λ hyperparameter is introduced to balance the bias and variance. The advantage function The definition of is based on the immediate reward, value function, and the advantage of the next state action: Among them, QoE i represents the instant reward, which is the viewing experience of the user of the kth video block, V φ (s) represents the total reward that can be obtained starting from state s under the current strategy φ, λ is the GAE parameter, s′ and a′ represent the next state and action in the current video transmission process respectively; Combining the strategy loss and value loss of PPO, the strategy loss measures the product of the probability ratio of the current strategy network and the old strategy network to select actions and the advantage estimate. The strategy loss is calculated by the following formula: where L(s,a,θ′,θ) is the loss function of the current state s, the selected action a, the current policy network parameters θ and the old policy network parameters θ′, π θ (a, s) refers to the probability of selecting action a in state s according to the current policy network θ, and is the advantage estimate of action a in state s based on the old policy network θ′, is a clipped version of the advantage function, where: where ∈ introduces a positive value for balancing policy update performance.

6. The method according to claim 1, characterized in that The adaptive retraining includes: using a counter to record the number of times the instantaneous reward of the mobile device for multiple consecutive data blocks is negative, and when the counter reaches a threshold δ, activating the online retraining process, and after the retraining process, the counter is reset to 0.

7. An adaptive super-resolution service device for mobile terminal video, characterized in that: include: The super-resolution model training module is configured to establish a multi-magnification super-resolution model adapted to the mobile terminal, use a large-scale data set for pre-training, and extract a specified frame for each target video to re-train the model. The trained super-resolution model is used to be deployed on the mobile terminal device; The intelligent decision-making agent model training module is configured to establish a reinforcement learning agent model that performs super-resolution multiples and bit rate decisions. The behavior a of the intelligent decision-making agent includes the bit rate b of the video block to be downloaded and the super-resolution multiple u. The environmental state information s obtained by the intelligent decision-making agent before making a decision includes the size of the next data block at different bit rates and video quality, the buffer size, and the time and energy consumption of loading the previous data block. The goal is to maximize the cumulative reward R for each action. i , the R i An intelligent decision-making agent model that is associated with the user's video viewing experience QoE and trained based on real data sets. The decisions made by the agent are transmitted to mobile devices. The adaptive retraining module is configured to continuously collect video stream data for each user, including immediate rewards, rebuffering time, and estimated bandwidth. When the system performance degrades and meets the specified conditions, the intelligent decision-maker uses the historical video blocks of the current user to update its policy network and uses the new policy network to select the next action, so that the video quality can adapt to different mobile devices and networks.

8. An intelligent decision-making and super-resolution service device, characterized in that: The device includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the adaptive super-resolution method for mobile video as described in any one of claims 1-6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the adaptive super-resolution method for mobile terminal video as described in any one of claims 1 to 6 of the present invention are implemented.

10. A video transmission system, characterized in that: It includes a content distribution network, a server, and a client. The server is configured to execute the adaptive super-resolution method for mobile video as described in any one of claims 1 to 6, or adopt the super-resolution service device as described in claim 7, or adopt the intelligent decision-making and super-resolution service device as described in claim 8, and send the super-resolution model and the decision made by the intelligent agent to the client. The client is a mobile device deployed with a super-resolution model trained by the server. The mobile device takes corresponding actions according to the decision of the intelligent decision-making body of the server, and pulls videos from the content distribution network for users to watch.