Online teaching text video super-resolution method based on svc and edge computing

Abstract

The application discloses an online teaching text video super-resolution method based on SVC and edge computing, which combines edge computing, SVC scalable video coding technology and super-resolution technology. The application uses the edge computing architecture to unload the video enhancement from the terminal to the edge cloud server, solving the problem of insufficient terminal computing power. The combination of the SVC scalable video coding mode and the super-resolution technology improves the video viewing quality under network fluctuations. Meanwhile, a text super-resolution model suitable for the characteristics of online teaching videos is proposed. The application can effectively improve the end user viewing quality of online teaching videos under network fluctuations, and improve the teaching and learning effect of teachers and students.

Description

Technical Field

[0001] This invention belongs to the field of video enhancement and network optimization, and involves SVC coding, edge computing and video enhancement techniques and methods. Background Technology

[0002] Schools and educational institutions have widely implemented online teaching. However, unstable network fluctuations can affect the continuity of classes and reduce the quality of online teaching. On the other hand, optimal downlink video quality is limited by uplink video quality. Without in-network video enhancement features, even if the network conditions and equipment capabilities at the receiving end allow for receiving better video quality, the downlink video stream can only adapt to the same or lower bitrate or frame rate as the uplink video stream, failing to maximize the user experience.

[0003] For network fluctuation issues, SVC (Scalable Video Coding) technology offers a solution. It is widely used in video transmission networks, enabling smooth video playback even under bandwidth constraints, and can adjust its encoding layers according to real-time network changes to achieve higher quality playback. SVC is well-compatible with common video codec standards such as VP9 and H.264, and can be customized for use in Google Chrome and WebRTC real-time communication technologies.

[0004] To address the issue that optimal downlink video quality is limited by uplink video quality, many works have proposed video enhancement through super-resolution, which can recover high-quality video even when uplink video quality is poor. LiveNAS and LiveSRVC, by combining super-resolution techniques with online training, have achieved high-quality live streaming to servers even with limited uplink bandwidth and computing power on the client side. While these works have achieved good results in real-time video super-resolution of natural scenes, they are not suitable for super-resolution of text-based videos. This is because text images are special images, possessing characteristics such as sharp edges, clear segmentation, and high contrast compared to natural images. Therefore, super-resolution methods for natural images, such as smoothing, will ignore the specific features of text. Currently, super-resolution work on small text images in natural scenes is relatively mature, but most of these works aim at accurately recognizing text content, using small image scene text datasets generated under various natural environments and human influences. The text outlines are often blurred, and the text shape, font, and background vary considerably. However, the text shape, font, and background of online teaching videos have little variation, and each frame is large with a large number of words. Therefore, directly applying the above methods to a single frame or the entire teaching video will lead to a decrease in super-resolution effect and a sharp increase in the time for super-resolution training and inference, which cannot meet the requirements of low latency for real-time video super-resolution.

[0005] Therefore, proposing a transmission optimization solution to enhance the quality of online education videos is a technical problem that urgently needs to be solved by those skilled in the art, so as to ensure high-quality, low-latency and stable transmission of online education videos. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and solve the problem of online teaching lag caused by network fluctuations and the limitation of optimal downlink video quality on uplink video quality. This invention can effectively improve the Quality of Experience (QoE), including video quality, smoothness, and low latency, even under network fluctuations, thereby improving the effectiveness of teaching and learning.

[0007] Technical solution of the present invention

[0008] A text-based video super-resolution method for online teaching based on SVC and edge computing is described below:

[0009] S1. A method combining edge cloud architecture, SVC scalable video coding, and super-resolution technology is proposed for video transmission and enhancement. Using an edge computing architecture offloads video enhancement from the terminal to an edge cloud server, solving the problem of insufficient terminal computing power; combining SVC and super-resolution technology improves video viewing quality even under network fluctuations.

[0010] S2. Constructing a text-based super-resolution method for online teaching video enhancement. For text-based online teaching videos, a real-time super-resolution method tailored to their characteristics is proposed. This method effectively addresses the issue that the optimal downlink video quality is limited by the uplink video quality through video enhancement.

[0011] The edge cloud architecture consists of a three-layer network: a core cloud, an edge cloud, and end users (including teacher and student end users). The core cloud, acting as the overall scheduling module for synchronous online teaching, provides unified management of online classrooms, offers accessible services to users, and orchestrates and manages resources for edge servers. The distributed edge cloud provides synchronous online teaching services to end users based on proximity, handling video stream reception and forwarding, online super-resolution model training, and super-resolution video enhancement. End users are the participants in the synchronous online teaching process, including teachers and students.

[0012] The steps of the method include:

[0013] 1) Teacher users register with the core cloud and open a classroom. Student users apply to join the same classroom with the core cloud using a unified password. Under the deployment of the core cloud, they join the corresponding edge cloud server nearby. After that, the video of the students is transmitted through this edge cloud server.

[0014] 2) After the class begins, the video stream from the teacher's end user is encoded by SVC at different layers (enhancement layer and base layer) and resolution according to the uplink network status and then transmitted to the edge server.

[0015] 3) The edge server receives the video stream and distributes it.

[0016] If no SVC video stream with enhancement layer is received, only the basic layer video stream is distributed to other edge cloud servers. If an SVC video stream with enhancement layer is received, the video stream is preprocessed, and the enhancement layer and basic layer are used as training sets to train the online teaching text video super-resolution model. The updated super-resolution model parameters and the SVC-encoded basic layer are then sent to other edge servers.

[0017] 4) Other edge servers update the model in real time, use the most recent model to perform real-time super-resolution inference on newly arriving basic layer video streams, and distribute the results to student users.

[0018] The described SVC (Scalable Video Coding) method, combined with super-resolution technology, determines the SVC coding method based on network status awareness and decides whether to incorporate super-resolution training based on the coding method. This optimizes video transmission latency, reduces backbone network transmission pressure, and avoids congestion and video playback stuttering. Real-time network status awareness is fundamental to video transmission decisions. The teacher-side user in this invention periodically captures packets to analyze uplink network bandwidth fluctuations, thereby determining the SVC coding method: when network conditions are poor, only the base layer is encoded; when network conditions are good, both the base layer and enhancement layers at different resolutions are encoded to support super-resolution training and inference requirements.

[0019] The SVC encoding method, which adapts to network conditions, allows the receiving edge server of the video source to receive videos of different resolutions: 270p, 360p, 540p, 720p, and 1080p. Therefore, the super-resolution model placed on the edge server in this invention can magnify the video by different factors (×4, ×3, ×2, ×1.5). When a 1080p video is received, it is added to the super-resolution training process; otherwise, only super-resolution inference is performed.

[0020] The super-resolution model deployed on an edge server is a text-video enhancement model for online teaching proposed in this invention. This model is a convolutional neural network model with residual blocks. It extracts features from the image through convolutional layers and activation layers. Five residual blocks (composed of a convolutional layer-BN layer-Mish activation layer-convolutional layer-BN layer-GRU layer-GRU layer) are stacked to extract deeper-level sequence-dependent features. Then, super-resolution upscaling at different scales is performed through a magnification module composed of convolutional layers and upsampling layers: for the ×1.5 model, the video frame is directly magnified to 1080p at input, with an upsampling factor of 1; for the ×2 and ×3 models, the upsampling factors are set to 2 and 3 respectively; for the ×4 super-resolution model, two magnification modules with a magnification factor of 2 are stacked. Finally, the Tanh function is used for activation. This invention also proposes a novel preprocessing and loss function.

[0021] The preprocessing method of the online teaching text video super-resolution model is as follows: traverse the video streams of high-resolution and low-resolution datasets, add 30 to the original RGB values ​​of color blocks with RGB values ​​less than 30, and add a binary mask channel to all frames sampled in the dataset. The binary mask is calculated using average grayscale and forms four channels (RGBM) as the input of the super-resolution model.

[0022] The image binary mask loss L proposed by the online teaching text video super-resolution model mask The calculation formula is as follows:

[0023]

[0024]

[0025] L mask The MSE loss is calculated using formula (2), where N is the number of pixels. M hr (x) and These are the binary mask values ​​of the original high-resolution frame HR and the high-resolution frame SR output by the super-resolution model, respectively. The binary mask is obtained by calculating the average gray level of the video frame. As shown in formula (1), it is the binary mask of the i-th pixel, where R, G, and B are the RGB values ​​of the i-th pixel, respectively.

[0026] The proposed online teaching text-to-video super-resolution method aligns with the text-based nature and practical needs of online teaching videos, enhancing the viewing experience for users.

[0027] The present invention has the following advantages over the prior art:

[0028] This invention proposes an edge computing architecture suitable for online education, which utilizes SVC encoding technology and introduces video enhancement methods. Compared with other existing technologies, it can better adapt to the network environment and achieve better video enhancement results. This effectively avoids the problem of poor viewing quality in online education videos caused by network fluctuations and the limitation of optimal downlink video quality on uplink video quality. Attached Figure Description

[0029] Figure 1 This is the overall design architecture diagram of the present invention;

[0030] Figure 2 This is a flowchart of the process of this invention;

[0031] Figure 3 This is a model diagram of the text-to-video super-resolution method for teaching proposed in this invention;

[0032] Figure 4 This is a comparison chart of the SSIM values ​​after super-resolution of the present invention and the TSRN and SRVC models. Detailed Implementation

[0033] To more clearly illustrate the technical solution of the present invention, a detailed description will be provided below in conjunction with the accompanying drawings and embodiments.

[0034] This invention builds Figure 1 The network architecture shown uses an Intel(R) Core(TM) i7-9700 as a cloud server, an Nvidia 3090ti and an Nvidia 3080ti as edge servers, and multiple Intel(R) Core(TM) i7-9700s as end users to simulate real educational and teaching scenarios.

[0035] The online teaching process based on the network architecture of this invention is as follows: Figure 2 As shown:

[0036] Teachers register with the core cloud and open a classroom. Students apply to join the same classroom through a unified password and join the corresponding edge cloud server based on the nearest edge cloud server deployed by the core cloud. After that, all videos of the students are transmitted through this edge cloud server.

[0037] After the class begins, the video stream from the teacher's end user is encoded using SVC based on the network conditions and then transmitted to the edge server. The edge server receives the video stream and distributes it. When it receives an SVC stream with a suitable enhancement layer, it adds it to the super-resolution training process and sends the super-resolution model to other edge servers. The other edge servers update the model in real time and perform real-time super-resolution inference on newly arriving videos.

[0038] During online teaching, various network fluctuations can occur. When the data transmission volume on the same link is large, it can reduce the allocable bandwidth for each process in the link, increase latency and packet loss rate, and cause lag. Therefore, this invention determines the SVC encoding format by judging the uplink network status, and the specific method is shown in Table 1:

[0039] Table 1. Correspondence between Network Status and Encoding Format

[0040]

[0041] In Table 1, when we detect that the network status is good or relatively good, we add the video stream transmitted from the client to the online training process of the edge server; when the network status is average or poor, we only perform super-resolution inference and do not participate in online training.

[0042] The super-resolution model structure proposed in this invention is as follows: Figure 3 As shown, firstly, convolutional layers are used to extract features from the image. Then, five residual blocks are stacked to extract deeper sequence-dependent features. Finally, super-resolution amplification is performed through convolutional layers and upsampling layers. The hidden layers of the ×2, ×3, and ×4 super-resolution models are set to 16 layers. For the ×1.5 super-resolution model (720p super-resolution to 1080p), the low-resolution frame is first upsampled to 1080p, and the hidden layers are set to 4 layers.

[0043] Online teaching videos, which are primarily text-based, often contain a large amount of black text. However, super-resolution of these black areas frequently results in color distortion, leading to poor video quality. This invention preprocesses color blocks with RGB values ​​less than 30 in all frames, adding 30 to their original RGB values, thus achieving a better super-resolution effect.

[0044] The overall loss function of the super-resolution model is shown in Equation (3), where ω1, ω2, and ω3 are respectively L GP L RGB L mask The weight of L. mask The image binary mask loss is calculated using the MSE loss, as shown in formula (2) above, where N is the number of pixels. and These are the binary mask values ​​for HR (the original high-resolution frame) and SR (the high-resolution frame output by the super-resolution model), respectively. The binary mask is obtained by calculating the average grayscale of the video frames. As shown in formula (1) above, this is the binary mask for the i-th pixel, where R, G, and B are the RGB values ​​of the i-th pixel, respectively. RGB The mean square error of RGB is calculated using the MSE loss, as shown in formula (4), where and These are the RGB values ​​of HR and SR, respectively. L GP This is the gradient prior loss, which can refine the boundary. It is calculated using L1 loss, as shown in equation (5). Represents the gradient field of the HR image. This represents the gradient field of the SR image.

[0045] Loss=ω1*L GP +ω2*L RGB +ω3*L mask (3)

[0046]

[0047]

[0048] To demonstrate the effectiveness of this invention, the super-resolution method was compared with TSRN and SRVC methods. A 10-minute segment of the experimental video was extracted, and training was performed by sampling 30 frames per second. TSRN and the models of this invention, based on pre-training, performed content-adaptive super-resolution on the experimental video. Since SRVC does not require pre-training, it directly performed content-adaptive super-resolution. To reduce the number of training iterations, when the PSNR value after super-resolution was less than 36 or the SSIM value was less than 0.99, a new frame was used to train the model, and the model was saved every 5 seconds. Figure 4 It can be observed that the super-resolution model of this invention outperforms other methods in the image reconstruction performance metric SSIM.

Claims

1. An online teaching text video super-resolution method based on SVC and edge computing, the method is as follows: S1, propose a method combining edge cloud architecture, SVC scalable video coding and super-resolution technology for video transmission and enhancement; S2, construct a text super-resolution method for online teaching video enhancement; The steps of the method include: 1) The teacher end user registers with the core cloud and opens the classroom, and the student end user applies to join the same classroom through a unified password and joins the corresponding edge cloud server under the deployment of the core cloud; 2) After the start of the class, the video stream of the teacher end user is SVC coded according to the uplink network state and transmitted to the edge cloud server by the teacher end user; the method for SVC coding of the video stream of the teacher end user according to the network state is: The teacher end user analyzes the network bandwidth fluctuation state in the uplink by periodic packet capture, and codes different levels and resolutions of SVC according to the network fluctuation, and only codes the base layer when the network state is bad, and codes the base layer and different resolution enhancement layers when the network state is good; The teacher end edge cloud server places four super-resolution models with different magnification scales, and the edge cloud server selects different video enhancement models with different magnification multiples according to the different SVC video streams sent by the teacher end user for training and inference; 3) The teacher end edge cloud server distributes the video stream after receiving it; 3.1) If the teacher end edge cloud server does not receive the SVC video stream with the enhancement layer, the base layer video stream coded by SVC is directly distributed to other edge cloud servers; 3.2) If the teacher end edge cloud server receives the SVC video stream with the enhancement layer, the video stream is preprocessed, the enhancement layer and the base layer are used as the training set to train the online teaching text video super-resolution model, and the updated model parameters and the base layer video stream coded by SVC are distributed to other edge cloud servers; The structure of the online teaching text video super-resolution model is as follows: First, a convolution layer and an activation layer are used to extract features from the image, then five residual blocks are stacked to extract deeper sequential dependence features, the residual block is composed of a convolution layer-BN layer-Mish activation layer-convolution layer-BN layer-GRU layer-GRU layer, then a magnification module composed of a convolution layer and an up-sampling layer is used for different scale super-resolution magnification: for the ×1.5 model, the video frame input is directly magnified to 1080p, and the up-sampling multiple is set to 1; for the ×2 and ×3 models, the up-sampling multiple is set to 2 and 3 respectively; for the ×4 super-resolution model, two magnification modules with a magnification multiple of 2 are stacked, and finally a Tanh function is used for activation; 4) Other edge cloud servers use the latest model to perform real-time super-resolution inference on newly arrived base layer video streams and deliver them to student end users.

2. The online teaching text video super-resolution method based on SVC and edge computing according to claim 1, characterized in that, The preprocessing process for training the online teaching text video super-resolution model using the enhancement layer includes: The enhancement layer in the online classroom video stream of the teacher end user is used as a high-resolution dataset, a low-resolution dataset is obtained by downsampling using an ffmpeg tool, video streams of the high-resolution and low-resolution datasets are traversed, color blocks with an RGB value less than 30 are added by 30 on the basis of original RGB values, and a binary mask channel is added to all frames sampled in the dataset, the binary mask is calculated using average gray, and four channels of RGBM are composed as inputs of a super-resolution model.

3. The online teaching text video super-resolution method based on SVC and edge computing according to any one of claims 1-2, characterized in that, The partial loss function of the online teaching text video super-resolution model is as follows: L mask Image binary mask loss, the calculation formula is shown as follows: (1) (2) L mask The MSE loss is calculated as shown in equation (2), where N is the number of pixels, ∇M hr (x) and ∇M sr (x) are the binary mask values of the original high-resolution frame HR and the high-resolution frame SR output by the super-resolution model, respectively. The binary mask is obtained by calculating the average grayscale of the video frame, as shown in equation (1) for the i-th pixel point, where R, G, and B are the RGB values of the i-th pixel point.

Images