Video playing method, mobile terminal and computer readable storage medium

Abstract

The invention discloses a video playing method, terminal equipment and a computer readable storage medium. The video playing method comprises the following steps: acquiring a video from a network camera or a network video recorder, acquiring a sub-code stream image and a main code stream image from the video, carrying out super-division reconstruction on the sub-code stream image according to a super-division model to obtain a first super-division image, calculating the image similarity between the first super-division image and the main code stream image, and obtaining a second super-division image; judging whether the image similarity exceeds a similarity threshold or not, if the image similarity is smaller than or equal to the similarity threshold, displaying the sub-code stream image on the terminal equipment, or updating the super-division model to obtain a new super-division model, performing super-division reconstruction on the sub-code stream image according to the new super-division model to obtain a second super-division image, and if not, performing super-division reconstruction on the sub-code stream image according to the second super-division image. And displaying the second super-resolution image on the terminal equipment. Thus, the terminal equipment can judge and correct the effect of the reconstructed super-resolution image in real time through a few main code stream images, and the problem that the quality of the playing content of the display interface is uncontrollable is solved.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a video playback method, a mobile terminal, and a computer-readable storage medium. Background Art

[0002] In the related art, monitoring equipment such as network cameras or network video recorders can receive requests sent by mobile devices and send the captured videos to mobile terminals for users to view. However, due to factors such as network bandwidth and the size of terminal device storage space, it is difficult for monitoring equipment to transmit complete main stream video data to mobile terminals in real time, so the mobile terminals often receive low-resolution sub-stream video data.

[0003] Furthermore, since the resolution of the factor stream video data is low, and some users expect to watch higher resolution videos on mobile terminals, a super-resolution reconstruction model can be deployed in some mobile terminals, so that the low-resolution image can be reconstructed into a high-resolution image through the locally deployed super-resolution reconstruction model.

[0004] However, in the related art, regardless of whether the reconstruction effect of the super-resolution reconstruction model on the low-resolution image is good or bad, the reconstructed image will be directly displayed in the playback interface. Once the reconstruction is wrong or the reconstruction effect is poor, it will greatly affect the user's viewing experience and even mislead the user's judgment in application fields such as criminal investigation. Summary of the invention

[0005] The present application provides a video playback method, a mobile terminal and a computer-readable storage medium.

[0006] The present application provides a video playback method, including:

[0007] Acquire a video from a network camera or a network video recorder, and acquire a sub-stream image and a main-stream image from the video;

[0008] Performing super-resolution reconstruction on the sub-stream image according to the super-resolution model to obtain a first super-resolution image;

[0009] Calculating the image similarity between the first super-resolution image and the main stream image;

[0010] Determining whether the image similarity exceeds a similarity threshold;

[0011] If the image similarity is less than or equal to the similarity threshold, the sub-stream image is displayed on the terminal device, or the super-resolution model is updated to obtain a new super-resolution model, and the sub-stream image is super-reconstructed according to the new super-resolution model to obtain a second super-resolution image, and the second super-resolution image is displayed on the terminal device.

[0012] Thus, in the embodiment of the present application, the terminal device can judge and correct the reconstructed super-resolution image effect in real time through a small number of main code stream images, thereby solving the problem of uncontrollable playback content quality of the display interface.

[0013] In an embodiment of the present application, the method further includes:

[0014] If the image similarity is greater than the similarity threshold, the first super-resolution image is displayed on the terminal device.

[0015] In the embodiment of the present application, the sub-stream images are acquired continuously, and the main stream images are acquired at preset frame intervals.

[0016] In the embodiment of the present application, the calculating the image similarity between the first super-resolution image and the main stream image includes:

[0017] Determining whether the sub-stream image and the main stream image are acquired at the same time;

[0018] If the sub-stream image and the main stream image are acquired at the same time, the image similarity between the first super-resolution image and the main stream image is calculated.

[0019] In an embodiment of the present application, updating the super-resolution model to obtain a new super-resolution model includes:

[0020] Calculate and update the super-resolution denoising weight coefficient according to the image similarity;

[0021] The super-resolution model is updated according to the updated super-resolution denoising weight coefficient to determine a new super-resolution model.

[0022] In an embodiment of the present application, updating the super-resolution model according to the updated super-resolution denoising weight coefficient to determine a new super-resolution model includes:

[0023] The new super-resolution model is determined according to the updated super-resolution denoising weight coefficient, the parameters of the super-resolution model with denoising, and the parameters of the super-resolution model without denoising.

[0024] In an embodiment of the present application, the method further includes:

[0025] In response to an operation of exiting the real-time preview playback interface, the second super-resolution image or the first super-resolution image is stopped from being displayed on the terminal device.

[0026] In an embodiment of the present application, the method further includes:

[0027] Determining whether the sub-stream image and the main stream image are acquired at the same time;

[0028] If the sub-stream image and the main stream image are acquired at the same time, the main stream image and the first super-resolution image are saved in pairs to an image pair list.

[0029] In the embodiment of the present application, the calculating the image similarity between the first super-resolution image and the main stream image includes:

[0030] Determining whether the end of the video has been reached;

[0031] If the end of the video is reached, the average of the similarities between the main stream image and the super-resolution image in each image pair in the image pair list is calculated to determine the image similarity.

[0032] In an embodiment of the present application, the method further includes:

[0033] If the image similarity is greater than the similarity threshold, all the first super-resolved images in the image pair list are combined into a video segment, and the video segment is displayed on the terminal device.

[0034] An embodiment of the present application further provides a mobile terminal, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned video playback method is implemented.

[0035] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by one or more processors, the above-mentioned video playback method is implemented.

[0036] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0038] Figure 1 It is a schematic diagram of the interaction scenario between the terminal device and the network camera and network video recorder;

[0039] Figure 2 A flowchart of video playback in the video playback method provided in an embodiment of the present application;

[0040] Figure 3 A flowchart of real-time preview in a video playback method provided in an embodiment of the present application;

[0041] Figure 4A schematic diagram of the super-resolution model training process provided for the implementation of this application;

[0042] Figure 5 For Figure 1 The hardware structure diagram of the mobile terminal is shown. DETAILED DESCRIPTION

[0043] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0044] For remote monitoring needs, users can set up monitoring devices such as IP cameras (IP CAMERA) and network video recorders (NVR) in the places they want to monitor, and receive the video stream data transmitted by the monitoring devices through mobile terminals such as mobile phones and tablets for viewing, thereby achieving the purpose of remote monitoring.

[0045] Furthermore, after the monitoring device performs a shooting action to generate high-resolution main stream video data and low-resolution main stream video data, due to limitations of factors such as the storage space and network bandwidth of the mobile terminal, it is difficult for the monitoring device to transmit the complete main stream video data to the mobile terminal in real time. In other words, the data received by the mobile terminal is often low-resolution sub-stream video data or a small amount of main stream video data.

[0046] Furthermore, since the resolution of the sub-stream video data is relatively low, and some users expect to watch higher-resolution videos on mobile terminals, some mobile terminals can be equipped with and use super-resolution reconstruction technology to meet user needs in related technologies. Specifically, super-resolution reconstruction refers to the technology of reconstructing a real, clear, and high-resolution image from a single frame or multiple consecutive frames of low-resolution images through image processing technology. It is widely used in fields such as intelligent monitoring that are limited by device power consumption, network bandwidth, and storage size, but are in urgent need of improving the playback quality of mobile terminals (such as mobile phones, tablets, and computers). Specifically, as Figure 1As shown, affected by the storage space size and network bandwidth of the mobile terminal 100, monitoring devices such as the network video recorder 200 or the network camera 300 are often unable to directly transmit the complete main stream video data to the mobile terminal 100 in real time for the user to view. The mobile terminal 100 can often only receive low-resolution sub-stream video data or a small amount of main stream video data from the network camera 200 or the network video recorder 300, and then use the super-resolution reconstruction technology to reconstruct the sub-stream image frame by frame into a super-resolution image (which has the same resolution as the main stream image), and finally directly display the super-resolution image in sequence on the playback interface to achieve high-definition or even ultra-clear playback quality.

[0047] Therefore, after receiving the sub-stream video data sent by the monitoring device, the mobile terminal 100 can use the super-resolution image reconstruction model to super-reconstruct each sub-stream in the sub-stream video data frame by frame to obtain a super-resolution image with a higher resolution than the sub-stream, or even a super-resolution image with the same resolution as the main stream, and then display the super-resolution image on the playback interface for the user to watch videos with high-definition or even ultra-clear playback quality.

[0048] In practical applications, super-resolution reconstruction of real images (i.e. images captured by monitoring equipment) is mostly achieved through deep learning. Super-resolution reconstruction solutions achieved through deep learning can be simply divided into two types: one is an image super-resolution reconstruction solution based on Generative Adversarial Networks (GAN), and the other is an image super-resolution reconstruction solution based on non-adversarial networks.

[0049] In the image super-resolution reconstruction scheme based on the generative adversarial network, the main idea is usually: first, construct the generator network, the discriminator network and the loss function (including reconstruction loss, perceptual loss and adversarial loss, etc.), then use the reconstruction loss to train the generator network in the first stage to obtain a good generator network, and then in the second stage, through the reconstruction loss, perceptual loss and adversarial loss, fine-tune the weights of the generator network and train the discriminator network to achieve adversarial training, and finally obtain the trained generator network model, which is the final image super-resolution reconstruction model. Finally, the low-resolution image is sent to the trained generator network, and the output of the model is the super-resolution reconstructed image.

[0050] It is understandable that the image super-resolution reconstruction scheme based on the generative adversarial network can ensure the richness of details and textures in the reconstructed image, achieve a good denoising effect, and ensure sufficiently high clarity, but there may be reconstruction information that is created out of nothing, resulting in artifacts that affect the visual perception.

[0051] In image super-resolution reconstruction based on non-adversarial networks, the main idea is usually: first, build a super-resolution network (i.e., generator network) and a reconstruction loss function, and then use the reconstruction loss to train the generator network to obtain a trained generator network, which is the final required image super-resolution reconstruction model.

[0052] It is understandable that the image super-resolution reconstruction scheme based on non-generative adversarial networks can ensure the authenticity of the reconstructed information, avoid the creation of reconstructed information out of thin air, and thus avoid the generation of artifacts that affect the visual experience. However, the details and textures in the reconstructed image are relatively scarce, and the clarity and denoising effects are poor.

[0053] It is understandable that the above two super-resolution reconstruction schemes have their own advantages, and both will perform poorly in certain scenarios. It is also understandable that in the actual application of any of the above two super-resolution reconstruction schemes, the low-resolution image is input into the model to obtain the model output image, and then the model output image is displayed to the user. However, if the image reconstruction effect is poor, it will seriously affect the user's perception. For users, when using it, they do not expect the effect to deteriorate after super-resolution. In line with the principle of "better to have less than better", it is necessary to further judge and correct the image after super-resolution reconstruction to ensure the effect of playback quality.

[0054] Therefore, in the field of intelligent monitoring, the following problems may exist in the real-time preview and video playback functions based on super-resolution reconstruction in mobile terminals:

[0055] 1. Unable to control the quality of the playback image. For example, in the video playback function, regardless of the quality of the super-resolution reconstruction, the reconstructed super-resolution image will be played on the playback interface. Super-resolution images with incorrect or poor reconstruction effects will greatly affect the user's viewing experience and may even mislead the user's judgment in special application fields such as criminal investigation.

[0056] Second, the quality of the playback image cannot be corrected. For example, in the real-time preview function, the image super-resolution reconstruction of the entire preview playback process is usually based on pre-set parameters, or in other words, the image super-resolution reconstruction of the entire preview playback process is usually based on fixed parameters in the model, and is not adjusted in real time according to the reconstruction effect at the current moment.

[0057] Based on the above problems that may be encountered, this embodiment provides a video playback method, which includes:

[0058] Acquire video from a network camera or network video recorder, and obtain sub-stream images and main-stream images from the video;

[0059] According to the super-resolution model, super-reconstruct the sub-stream image to obtain a first super-resolution image;

[0060] Calculating the image similarity between the first super-resolution image and the main stream image;

[0061] Determine whether the image similarity exceeds a similarity threshold;

[0062] If the image similarity is less than or equal to the similarity threshold, the sub-stream image is displayed on the mobile terminal 100, or the super-resolution model is updated to obtain a new super-resolution model, and the sub-stream image is super-reconstructed according to the new super-resolution model to obtain a second super-resolution image, and the second super-resolution image is displayed on the mobile terminal 100;

[0063] If the image similarity is greater than the similarity threshold, the first super-resolution image is displayed on the mobile terminal 100 .

[0064] Specifically, in this embodiment, the video may be sourced from Figure 1 The monitoring devices such as the network camera 200 or the video recorder 300 shown may also be derived from a database storing videos, a computer memory, or other electronic devices, etc. This embodiment only takes the monitoring device as an example to specifically explain the implementation of each step of the video playback method described in the present application, wherein the monitoring device includes, but is not limited to, the network camera 200 or the video recorder 300.

[0065] In this embodiment, the mobile terminal 100 can realize playback functions such as video playback through the video sent by the monitoring device. Specifically, the monitoring device continuously and uninterruptedly generates sub-stream images and main-stream images, and continuously and uninterruptedly sends each sub-stream image to the mobile terminal 100, and sends the main-stream image to the mobile terminal 100 at a certain frame interval. Among them, the frame interval can be set to any integer according to actual conditions. When it is set to 10, the monitoring device sends one main-stream image to the mobile terminal 100 every 10 frames of main-stream images, and then after continuously sending 10 sub-stream images to the mobile terminal 100, when sending the 11th sub-stream image, it will also send one main-stream image to the mobile terminal 100 at the same time. In other words, in this embodiment, among the frame images that the mobile terminal 100 can obtain, the sub-stream images are continuously obtained, and the main-stream images are obtained at a preset frame interval.

[0066] For a more clear description of this embodiment, please refer to Figure 1 and Figure 2 , Figure 2 The flowchart of the video playback in the video playback method provided in the embodiment of the present application is as follows Figure 2 As shown, the video playback method in the video playback method provided in this embodiment is performed according to the following steps:

[0067] Step S201: Load the super-resolution model model deployed in the mobile terminal, perform super-resolution reconstruction on the sub-stream image, and obtain a first super-resolution image with the same resolution as the main stream image.

[0068] Specifically, in this embodiment, the mobile terminal 100 is equipped with a surveillance video playback function, or in other words, the mobile terminal 100 can display the video captured by surveillance devices such as the network camera 200 or the network video recorder 300 on a video playback interface for the user to watch.

[0069] Since the realization of functions such as surveillance video playback depends on the super-resolution model model, the mobile terminal 100 can pre-load the locally deployed super-resolution model model so that after receiving the sub-stream image sent by the monitoring equipment such as the network camera 200 or the network video recorder 300, the received sub-stream image can be super-reconstructed to obtain an image with the same resolution as the main stream image, that is, the first super-resolution image.

[0070] Step S202: according to the user's playback requirement, find the corresponding video from the network camera 200 or the network video recorder 300, and continuously obtain the sub-stream image from the video and obtain the main stream image at a certain frame interval.

[0071] Specifically, in this embodiment, the mobile terminal 100 responds to the surveillance video playback instruction triggered by the user, determines the video that the user desires to play back from the video captured by the surveillance device such as the network camera 200 or the network video recorder 300, and extracts continuous and uninterrupted sub-stream images and main stream images extracted at a certain frame interval from the video. The frame interval can be any integer such as 10.

[0072] Specifically, the user can select a desired video from various videos captured by monitoring devices such as the network camera 200 or the network video recorder 300 through the mobile terminal 100 according to actual needs. Accordingly, the mobile terminal 100 can continuously obtain the sub-stream image of the video from the monitoring device such as the network camera 200 or the network video recorder 300 according to the user selection result, and obtain the main stream image of the video at a certain frame interval.

[0073] Step S203: All video images (including sub-stream images and main-stream images) acquired at the current moment are transmitted to the mobile terminal.

[0074] Specifically, in this embodiment, the monitoring device sends the continuous and uninterrupted sub-stream images corresponding to the video that the user wants to play back to the mobile terminal 100, extracts multiple main stream images at a certain frame interval from the continuous and uninterrupted main stream images corresponding to the video, and sends the extracted multiple main stream images to the mobile terminal 100. For example, when the frame interval is set to 10, after the monitoring device continuously sends 10 sub-stream images to the mobile terminal 100, it will also send 1 main stream image to the mobile terminal 100 when sending the 11th sub-stream image.

[0075] Step S204: Based on the super-resolution model model, super-reconstruct the sub-stream image to obtain a first super-resolved image with the same resolution as the main stream image.

[0076] Specifically, in this embodiment, when the mobile terminal 100 receives a sub-stream image at the current moment, it can use the previously loaded super-resolution model model to infer and reconstruct the sub-stream image received at the current moment, so as to obtain an image with the same resolution as the main stream image, that is, the first super-resolution image, and save the first super-resolution image to prepare in advance for the subsequent possible super-resolution effect display.

[0077] Step S205: Determine whether the main stream image is received at the same time. If the main stream image is received at the same time, execute step S206; if the main stream image is not received at the same time, return to step S204.

[0078] Specifically, in this embodiment, when the terminal device 100 receives an image sent by the monitoring device at any time, it can also determine whether the image received at the current time is only one sub-stream image, or a combination of one sub-stream image and one main stream image.

[0079] Furthermore, if the image received at the current moment is only one sub-stream image, the terminal device 100 may wait for the next image sent by the monitoring device to perform super-resolution reconstruction on the next received sub-stream image through the super-resolution model model, that is, return to step S204.

[0080] Step S206: The first super-resolution image and the main stream image at the current moment are sequentially stored in pairs in an image pair list.

[0081] Specifically, in this embodiment, if at the current moment, in addition to receiving one sub-stream image, one main stream image is also received, the terminal device 100 saves the "main stream image at the current moment" and the "first super-resolution image obtained by super-reconstructing the 'sub-stream image at the current moment'" in a pair to the image pair list. That is, in some embodiments of the present application, the video playback method further includes: determining whether the sub-stream image and the main stream image are obtained at the same time; if the sub-stream image and the main stream image are obtained at the same time, saving the main stream image and the first super-resolution image in a pair to the image pair list.

[0082] Step S207: Determine whether the video has reached the end. If the image frame sent by the monitoring device has reached the end of the video, execute step S208; if the image frame sent by the monitoring device has not reached the end of the video, return to step S204.

[0083] Specifically, in this embodiment, after the terminal device 100 completes the super-resolution processing of a sub-stream image to obtain the first super-resolution image, or saves the main stream image and the first super-resolution image in pairs to the image pair list, it can determine whether the image frame sent by the monitoring device this time is the end of the video, that is, the last frame of the video.

[0084] Furthermore, if the image frame sent by the monitoring device this time is not the end of the video, the terminal device 100 can wait for the next image sent by the monitoring device to perform super-resolution reconstruction and other operations on the next received sub-code stream image through the super-resolution model model. In other words, repeat steps S204-S207.

[0085] Step S208: Calculate the average image similarity of all first super-resolved images and main stream images in the image pair list.

[0086] Specifically, in this embodiment, if the end of the video is reached, the terminal device 100 can calculate the average image similarity SSIM' between each first super-resolution image in the image pair list and the main stream image corresponding to each first super-resolution image according to the first super-resolution image-main stream image pair list collected in pairs in step S206. That is, in some embodiments of the present application, the video playback method further includes: determining whether the end of the video is reached, and if the end of the video is reached, calculating the average similarity between the main stream image and the first super-resolution image in each image pair in the image pair list to determine the image similarity.

[0087] It is understandable that common image similarity methods include structural similarity (SSIM), peak signal-to-noise ratio (PSNR) and mean square error (MSE). In this embodiment, the image similarity to be calculated can be one similarity calculation method among the above structural similarity, peak signal-to-noise ratio and mean square error, or a combination of multiple similarity calculation methods.

[0088] Meanwhile, in order to clearly explain this embodiment, the following description will be made by taking the image similarity of structural similarity as an example. That is, in this embodiment, the average image similarity can be calculated by the following formula, namely:

[0089]

[0090] Where SSIM' represents the average image similarity; N represents the number of image pairs in the image pair list; x i and i denote the first super-resolution image and the main stream image of the i-th pair respectively; and Respectively represent the i-th image pair x i and i The average pixel value of and Respectively represent the i-th image pair x i and i The standard deviation of represents the i-th image pair x i and i covariance; c1 and c2 are constants used to avoid errors caused by the denominator being 0; SSIM′∈[0,1], the larger the value of SSIM′, the smaller the deviation between each first super-resolution image in the number of image pairs and its corresponding main stream image, and thus the better the quality of the first super-resolution image.

[0091] Step S209: Determine whether the average image similarity is greater than the similarity threshold. If the average image similarity is not greater than (less than or equal to) the similarity threshold, execute step S210; if the average image similarity is greater than the similarity threshold, execute step S211.

[0092] Specifically, in this embodiment, after the terminal device 100 calculates the above average image similarity SSIM, it can determine whether the average image similarity SSIM' is greater than or equal to the set similarity threshold, thereby determining whether the reconstructed super-resolution effect meets the playback requirements.

[0093] Step S210: directly playing the original sub-stream video on the video playback interface.

[0094] Specifically, in this embodiment, if the average similarity SSIM' is less than the set similarity threshold, it indicates that the super-resolution effect of the video reconstructed by the super-resolution model does not meet the playback requirements. Therefore, in order to avoid the situation where the user's viewing experience may be affected when the super-resolution video of poor quality is displayed to the user, the original sub-stream video is directly played on the video playback interface.

[0095] Step S211: combining the first super-resolution images of the saved continuous frames into a video clip, and playing it on the video playback interface.

[0096] Specifically, in this embodiment, if the average image similarity SSIM' is greater than or equal to the set similarity threshold, it indicates that the reconstructed super-resolution effect meets the playback requirements, and then the previously saved multiple consecutive first super-resolution images can be combined into a video clip and played on the video playback interface. That is, in certain embodiments of the present application, if the image similarity is greater than the similarity threshold, the saved multiple consecutive frames of the first super-resolution image are combined into a video clip, and the video clip is displayed on the video playback interface.

[0097] At this point, based on step S201 to step S211, the terminal device 100 can use a small number of main stream images to achieve the purpose of controlling the playback quality of the video playback interface.

[0098] Thus, in this embodiment, a small number of main stream images can be used to achieve the purpose of controlling the playback quality of the video playback interface, thereby improving the user experience and avoiding the situation where the quality of the played images is discontinuous.

[0099] In some embodiments of the present application, the mobile terminal 100 can implement a monitoring image real-time preview function based on the received main stream image and sub stream image. To clearly explain the execution logic of the monitoring image real-time preview function in this embodiment, please refer to Figure 3 , Figure 3 A flowchart of real-time preview in the video playback method provided in the embodiment of the present application, that is, Figure 3 As shown, the real-time preview method in the video playback method provided in this embodiment is performed according to the following steps:

[0100] Step S301: Obtain a super-resolution model model=α×model1+(1-α)×model2 according to the default super-resolution denoising weight coefficient α, the super-resolution model model1 with denoising, and the super-resolution model model2 without denoising.

[0101] Specifically, in this embodiment, the terminal device 100 can first fuse the weights of model1 and model2 through a deep network interpolation method according to the model parameters of the super-resolution model model1 with denoising, the model parameters of the super-resolution model model2 without denoising, and the default super-resolution denoising weight coefficient α, to obtain the super-resolution model model. The deep network interpolation method refers to two models with exactly the same network structure but different network weights, and all corresponding weight parameter values ​​of the two models are updated one by one according to the linear interpolation method, so as to obtain a new model after weight fusion.

[0102] In an example, the process of obtaining the super-resolution model model can be shown as follows:

[0103] model=α×model1+(1-α)×model2

[0104] In an example, the super-resolution denoising weight coefficient is denoted by α, α∈[0,1], and the default value is 1.

[0105] In an example, the specific method of obtaining the super-resolution model model1 with denoising and the super-resolution model model2 without denoising can be found in Figure 4 , Figure 4 Schematic diagram of the super-resolution model training process provided for the implementation of this application. Specifically, the generator network fine-tuned in the second stage of the super-resolution model is used as an adversarial super-resolution model, denoted as super-resolution model model1 with denoising. At the same time, in order to more conveniently and uniformly set super-resolution reconstruction models with different weights, the generator network trained in the first stage of the super-resolution model can be directly used as a non-adversarial super-resolution model, denoted as super-resolution model model1 without denoising. It can be understood that the super-resolution model model2 with denoising and the super-resolution model model2 without denoising represent the image super-resolution reconstruction model based on the adversarial network and the image super-resolution reconstruction model based on the non-adversarial network, respectively.

[0106] Step S302: continuously acquiring sub-stream images from the network camera and acquiring main stream images at a certain frame interval.

[0107] Specifically, in this embodiment, the mobile terminal 100 can continuously obtain the sub-stream image of the video to be previewed from the monitoring device such as the network camera 200 or the network video recorder 300, and obtain the main stream image of the video at a certain frame interval.

[0108] Step S303: All images (including sub-stream images and main-stream images) acquired at the current moment are transmitted to the mobile terminal.

[0109] Specifically, the monitoring device can continuously and uninterruptedly generate sub-stream images and main-stream images, and continuously and uninterruptedly send each sub-stream image to the mobile terminal 100, and send the main-stream image to the mobile terminal 100 at a certain frame interval. The frame interval can be set to any integer according to actual conditions. When it is set to 10, the monitoring device sends one main-stream image to the mobile terminal 100 every 10 main-stream images. After the monitoring device continuously sends 10 sub-stream images to the mobile terminal 100, when sending the 11th sub-stream image, it will also send one main-stream image to the mobile terminal 100 at the same time.

[0110] Step S304: Based on the super-resolution model model, super-reconstruct the sub-stream image to obtain a first super-resolved image with the same resolution as the main stream image.

[0111] Specifically, in this embodiment, when the mobile terminal 1000 receives the sub-stream image sent by the monitoring device, the super-resolution model model can be used to infer and reconstruct the sub-stream image to obtain a super-resolution reconstructed image with the same resolution as the main stream image, that is, the first super-resolution image.

[0112] Step S305: Determine whether the main stream image is received at the same time. If the main stream image is received at the same time, execute step S306; if the main stream image is not received at the same time, execute step S310.

[0113] Specifically, in this embodiment, when the terminal device 100 receives an image sent by the monitoring device at any time, it can also determine whether the image received at the current time is only one sub-stream image, or a combination of one sub-stream image and one main stream image.

[0114] Step S306: Calculate the image similarity between the first super-resolution image and the main stream image.

[0115] Specifically, if the mobile terminal 100 receives a main stream image in addition to a sub-stream image at the current moment, the mobile terminal 100 can perform quality control and correction on the first super-resolution image at the current moment according to the main stream image. Specifically, the mobile terminal 100 calculates the image similarity based on the first super-resolution image and the main stream image at the current moment. That is, in this embodiment, the above-mentioned step of calculating the image similarity between the first super-resolution image and the main stream image includes: determining whether the sub-stream image and the main stream image are obtained at the same time; if the sub-stream image and the main stream image are obtained at the same time, calculating the image similarity between the first super-resolution image and the main stream image.

[0116] It is understandable that common image similarity methods include structural similarity (SSIM), peak signal-to-noise ratio (PSNR) and mean square error (MSE). In this embodiment, the image similarity to be calculated can be a similarity calculation method or a combination of multiple similarity calculation methods among the above-mentioned structural similarity, peak signal-to-noise ratio and mean square error. In one example, the mobile terminal 100 uses the calculation method of structural similarity SSIM, and therefore, the image similarity between the first super-resolution image and the main stream image can be calculated by the following formula, namely:

[0117]

[0118] Where SSIM represents the image similarity between the first super-resolution image and the main stream image; x and y represent the first super-resolution image and the main stream image respectively; μ x and μ y Represents the average pixel value of x and y respectively; σ x and σ y Represent the standard deviation of x and y respectively; σ xy Represents the covariance of x and y; c1 and c2 are constants used to avoid errors caused by the denominator being 0; SSIM(x,y)∈[0,1], the larger the SSIM value, the smaller the deviation between the first super-resolution image and the main stream image, in other words, the better the quality of the first super-resolution image.

[0119] Step S307: Determine whether the image similarity is less than the similarity threshold. If the image similarity is less than the similarity threshold, execute step S308; if the image similarity is not less than (greater than or equal to) the similarity threshold, execute step S310.

[0120] Specifically, in this embodiment, the terminal device 100 can determine whether the image similarity SSIM between the first super-resolution image and the main code stream image is less than a preset similarity threshold after calculating the image similarity SSIM, so as to determine whether the first super-resolution image meets the real-time preview requirements and whether it needs to be corrected, and determine whether the parameters of the super-resolution model model need to be updated.

[0121] Step S308: Calculate and update the super-resolution denoising weight coefficient α' according to the image similarity to update the super-resolution model model'=α'×model1+(1-α')×model2.

[0122] Specifically, in this embodiment, if the image similarity SSIM is less than a preset similarity threshold, it indicates that there is a large deviation between the real image (i.e., the main stream image) and the first super-resolution image output by the super-resolution model model with the current model parameters, or in other words, the distortion problem of the first super-resolution image is more serious.

[0123] And, since the super-resolution denoising weight coefficient α is 1 by default, the initial super-resolution model model can be regarded as the super-resolution model model1 with denoising, and it can be considered that the current scene is not suitable for using a super-resolution model with too strong denoising strength. Therefore, the size of α can be reduced to achieve the correction of the model and the image. That is, in this embodiment, the above-mentioned steps of updating the super-resolution denoising weight coefficient, updating the super-resolution model, and determining a new super-resolution model include: determining a new super-resolution model according to the updated super-resolution denoising weight coefficient, the parameters of the super-resolution model with denoising, and the parameters of the super-resolution model without denoising.

[0124] Specifically, the super-resolution denoising weight coefficient α′ can be updated according to the image similarity SSIM calculated at the current moment and the pre-constructed function F. The updating process can be shown as follows, namely:

[0125] α′=F(SSIM)

[0126] The function F can be a linear function in the form of α′=a×SSIM+b (a and b are both constants), or a linear function in the form of α′=a×SSIM 2 +b (a and b are both constants) is a nonlinear function of this form.

[0127] Next, the updated super-resolution denoising weight coefficient α′ is used to update the parameters of the super-resolution model model to obtain a new super-resolution model model′.

[0128] Among them, the acquisition process of the super-resolution model model' can be expressed by the following formula, namely:

[0129] model'=α′×model1+(1-α′)×model2

[0130] Step S309: Use the new super-resolution model to perform super-resolution reconstruction on the sub-stream image to obtain a second super-resolution image.

[0131] Specifically, in this embodiment, the terminal device 100 can re-super-reconstruct the sub-stream video image at the current moment according to the super-resolution model model' after parameter update, and output a corrected super-resolution image, that is, a second super-resolution image.

[0132] Step S310: Displaying the super-resolution image at the current moment on the real-time preview playback interface.

[0133] Specifically, in this embodiment, after obtaining the second super-resolution image, the terminal device 100 may display the second super-resolution image on the real-time preview playback interface. That is, in this embodiment, the video playback method further includes: if the image similarity is less than or equal to the similarity threshold, calculating and updating the super-resolution denoising weight coefficient according to the image similarity; updating the super-resolution model according to the updated super-resolution denoising weight coefficient, and determining a new super-resolution model; super-reconstructing the sub-stream image according to the new super-resolution model to obtain the second super-resolution image; and displaying the second super-resolution image.

[0134] In addition, in this embodiment, if in the determination process shown in S307, the image similarity SSIM is greater than or equal to the preset similarity threshold, it indicates that the first super-resolution image meets the real-time preview requirement, so there is no need to update the parameters of the super-resolution model model, and there is no need to correct the first super-resolution image, so the first super-resolution image at the current moment is directly displayed on the real-time preview playback interface. That is, in this embodiment, the video playback method also includes: if the image similarity is greater than the similarity threshold, then displaying the first super-resolution image.

[0135] Step S311: Whether to exit the real-time preview playback interface.

[0136] Specifically, in this embodiment, if the terminal device 100 receives a real-time preview playback interface exit instruction triggered by the user, the terminal device 100 ends the entire process. That is, in this embodiment, the video playback method further includes: in response to the operation of exiting the real-time preview playback interface, stopping displaying the second super-resolution image or the first super-resolution image on the terminal device. On the contrary, if the terminal device 100 does not receive a real-time preview playback interface exit instruction triggered by the user, the terminal device 100 repeats the above steps S302-S311.

[0137] Thus, in this embodiment, the mobile terminal 100 can judge and correct the reconstructed super-resolution image effect in real time through a small number of main stream images, thereby solving the problem of uncontrollable quality of the playback content on the display interface.

[0138] It is also understandable that based on Figure 2 and Figure 3 According to the steps shown, when the mobile terminal 100 uses the playback functions such as real-time preview or video playback, the playback image quality reconstructed using the super-resolution reconstruction technology can be guaranteed, thereby improving the user's viewing experience. In addition, by introducing a small number of main stream images, the image super-resolution effect can be judged and corrected to improve the image quality of the playback interface such as real-time preview or video playback.

[0139] See also Figure 5This embodiment further provides a mobile terminal 100, which includes, but is not limited to, a memory 110 and a processor 120. The video playback method of this embodiment can be implemented by the mobile terminal 100 of this embodiment.

[0140] The memory 110 is used to store computer programs. The memory 110 may include a high-speed RAM memory, or may also include a non-volatile memory, such as at least one disk memory. The computer program may include computer executable instructions.

[0141] The processor 120 is used to execute a computer program to implement the various method steps of the video playback method of the embodiment of the present application.

[0142] The processor 120 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0143] This embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, the steps of the above-mentioned video playback method are implemented.

[0144] In the description of this specification, the descriptions with reference to the terms "specifically", "further", "particularly", "understandably", etc. are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not intended to refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0145] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0146] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A video playback method, characterized in that: include: Acquire a video from a network camera or a network video recorder, and acquire a sub-stream image and a main-stream image from the video; Performing super-resolution reconstruction on the sub-stream image according to the super-resolution model to obtain a first super-resolution image; Calculating the image similarity between the first super-resolution image and the main stream image; Determining whether the image similarity exceeds a similarity threshold; If the image similarity is less than or equal to the similarity threshold, the sub-stream image is displayed on the terminal device, or the super-resolution model is updated to obtain a new super-resolution model, and the sub-stream image is super-reconstructed according to the new super-resolution model to obtain a second super-resolution image, and the second super-resolution image is displayed on the terminal device.

2. The video playback method according to claim 1, characterized in that: The method further comprises: If the image similarity is greater than the similarity threshold, the first super-resolution image is displayed on the terminal device.

3. The video playback method according to claim 1 or 2, characterized in that: The sub-stream images are acquired continuously, and the main stream images are acquired at preset frame intervals.

4. The video playback method according to claim 1 or 2, characterized in that: The calculating the image similarity between the first super-resolution image and the main stream image includes: Determining whether the sub-stream image and the main stream image are acquired at the same time; If the sub-stream image and the main stream image are acquired at the same time, the image similarity between the first super-resolution image and the main stream image is calculated.

5. The video playback method according to claim 1, characterized in that: The updating of the super-resolution model to obtain a new super-resolution model comprises: Calculate and update the super-resolution denoising weight coefficient according to the image similarity; The super-resolution model is updated according to the updated super-resolution denoising weight coefficient to determine a new super-resolution model.

6. The video playback method according to claim 5, characterized in that: The updating of the super-resolution model according to the updated super-resolution denoising weight coefficient to determine a new super-resolution model comprises: The new super-resolution model is determined according to the updated super-resolution denoising weight coefficient, the parameters of the super-resolution model with denoising, and the parameters of the super-resolution model without denoising.

7. The video playback method according to claim 1 or 2, characterized in that: The method further comprises: In response to an operation of exiting the real-time preview playback interface, the second super-resolution image or the first super-resolution image is stopped from being displayed on the terminal device.

8. The video playback method according to claim 2, characterized in that: The method further comprises: Determining whether the sub-stream image and the main stream image are acquired at the same time; If the sub-stream image and the main stream image are acquired at the same time, the main stream image and the first super-resolution image are saved in pairs to an image pair list.

9. The video playback method according to claim 8, characterized in that: The calculating the image similarity between the first super-resolution image and the main stream image includes: Determining whether the end of the video has been reached; If the end of the video is reached, the average of the similarities between the main stream image and the super-resolution image in each image pair in the image pair list is calculated to determine the image similarity.

10. The video playback method according to claim 9, characterized in that: The method further comprises: If the image similarity is greater than the similarity threshold, all the first super-resolved images in the image pair list are combined into a video segment, and the video segment is displayed on the terminal device.

11. A mobile terminal, characterized in that: The device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the video playback method according to any one of claims 1 to 10 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the video playback method according to any one of claims 1 to 10 is implemented.

Images