Terminal, program, and information processing method for displaying low-resolution images as high-resolution images

Abstract

To provide a video distribution system and the like that can be installed in a smartphone terminal or the like, which has lower performance such as a CPU and a memory than an electronic calculator such as a computer.SOLUTION: A video distribution system 1 includes an acquisition unit 10 that acquires low-resolution image data transmitted from a server, a processing unit 13 that generates high-resolution image data by executing predetermined processing on the acquired low-resolution image data through a predetermined AI model 20, a reconstruction unit 14 that reconstructs data of the generated high-resolution image, and a display unit 15 that displays the reconstructed high-resolution image data on the screen of a client terminal 6.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a video distribution system that has been improved as a lightweight model so that users can view live video (video distributed in real time) on, for example, a smartphone terminal. [Background technology]

[0002] An image transmission / reception system is disclosed that can efficiently compress the transmission bandwidth and restore images with a resolution close to the original image while reducing the burden on the operator, in order to reduce the load on the transmission path by reducing the distribution capacity and improving the quality of the images viewed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2020-109897 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the invention of Patent Document 1 has a problem in that the image transmission / reception system has a large number of parameters, making it difficult for users to view video in real time on smartphone terminals or the like.

[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a video processing system that can be installed in a smartphone terminal or the like, which has lower performance of a CPU (Central Processing Unit) and memory (RAM (Random Access Memory)) than an electronic calculator such as a computer, and that can be viewed in real time. Terminal, program and information processing method The purpose is to provide [Means for solving the problem]

[0006] According to the present invention, there is provided a terminal that receives low-resolution image data from a server, comprising: decomposition means for decomposing the low-resolution image data into frames; Given parameters (d, 3, 1)( d=18, 10, 5 or 3 a processing means for arranging multiple FSRCNNs (Fast Super-Resolution Convolutional Neural Networks) in parallel, processing multiple frames of low-resolution images simultaneously, and generating high-resolution image data for each frame; a reconstruction means for reconstructing data of a plurality of high resolution images according to a reproduction position; a display means for displaying the reconstructed high-resolution image data on a screen; The present invention is characterized by having the following.

[0009] According to the present invention, there is provided a program for causing a computer installed in a device that receives low-resolution image data from a server to function, the program comprising: decomposition means for decomposing the low-resolution image data into frames; Given parameters (d, 3, 1)( d=18, 10, 5 or 3 ) a processing means for arranging a plurality of FSRCNNs in parallel, processing a plurality of frames of low-resolution images simultaneously, and generating high-resolution image data for each frame; a reconstruction means for reconstructing data of a plurality of high resolution images according to a reproduction position; a display means for displaying the reconstructed high-resolution image data on a screen; The present invention is characterized by the fact that it makes a computer function by using the above-mentioned method.

[0010] An information processing method for a device that receives low-resolution image data from a server, comprising: The device is A first step of decomposing the low-resolution image data into frames; Given parameters (d, 3, 1)( d=18, 10, 5 or 3 ) in parallel, process multiple frames of low-resolution images simultaneously, and generate high-resolution image data for each frame. a third step of reconstructing the data of the plurality of high-resolution images according to the reproduction position; The fourth step is to display the reconstructed high-resolution image data on the screen. The present invention is characterized by carrying out the following. [Effects of the Invention]

[0011] Book According to the invention, high-resolution image data is generated within the client terminal from low-resolution image data transmitted and acquired from the server, thereby reducing the amount of bandwidth required when viewing high-resolution images on the client terminal. In addition, since the AI ​​model can be installed on the client terminal, real-time viewing is possible on the client terminal.

[0012] Also, this According to the invention, by arranging multiple AI models in parallel, high-resolution image data can be generated from low-resolution image data at high speed.

[0013] Furthermore, this According to the invention, by using a lightweight FSRCNN model, high-resolution image data can be generated from low-resolution image data with high reproducibility. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a functional block diagram showing the overall structure of a distribution system according to an embodiment of the present invention; [Figure 2] FIG. 1 is a diagram showing the structure of FSRCNN, which is an example of an AI model according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram showing a comparison between the first-stage learning results and the second-stage learning results. [Figure 4] 1 is a table summarizing verification results of each model of FSRCNN according to an embodiment of the present invention. [Figure 5]Among the FSRCNN models according to the embodiment of the present invention, model 56 (FSRCNN(56,12,4)) and model 18 (FSRCNN(18,3,1)) are compared in terms of the filter size, number of filters, number of channels, input images, and number of parameters for the four convolutional layers and one deconvolutional layer. [Figure 6] 3 is a flowchart showing a processing procedure in the video distribution system according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] An embodiment of the present invention will be described with reference to FIGS. 1 to 6. FIG.

[0016] FIG. 1 is a functional block diagram showing the overall structure of a distribution system 1 according to an embodiment of the present invention.

[0017] An example of viewing an online seminar in real time will be described below. A lecturer 2 uses a camera attached to an electronic computer 3 such as a computer to record video of the seminar.

[0018] The network 7 is a communication medium that wirelessly connects the computer 3 and the server 4. The network 8 is a communication medium that wirelessly connects the server 4 and the client terminal 6. The networks 7 and 8 may be of any communication format or medium as long as they can send and receive image data.

[0019] The client terminal 6 may be a smartphone terminal, a so-called wearable terminal such as a smart watch or smart glasses, or any other portable multi-function terminal that can use application software. In Fig. 1, a smartphone terminal will be used as an example.

[0020] Image data of the seminar being filmed is sent from the computer 3 to the server 4. The image data of the seminar being filmed is high-resolution image data.

[0021] Although an example using an electronic calculator such as a computer has been described, the seminar can also be filmed using a video camera with a communication function and image data of the filmed seminar can be sent to server 4; the present invention is not limited to this.

[0022] The server 4 generates low-resolution image data from the received seminar image data, and performs encoding processing on the low-resolution image data in order to multicast it to the client terminals 6. When performing encoding processing, the image resolution can be specified.

[0023] The acquisition unit 10 acquires the encoded low-resolution image data by multicasting it from the server 4 to the client terminal 6 of the user 5. Note that in Fig. 1, only one user 5 is shown and only one client terminal 6 is shown, but in reality there are at least as many users and client terminals as the number of users participating in the online seminar.

[0024] The decomposing unit 11 receives the low-resolution image data acquired by the acquiring unit 10 and decomposes the low-resolution image data frame by frame in the order in which they were acquired. The frame size to be decomposed will be described later.

[0025] The temporary storage unit 12 temporarily stores the data of the low-resolution image decomposed by the decomposing unit 11.

[0026] The processing unit 13 receives the decomposed low-resolution image data from the temporary storage unit 12, and then generates high-resolution image data in the AI ​​model 20. A plurality of AI models 20 are arranged in parallel in the processing unit 13. When the AI ​​model 20 completes generation of high-resolution image data (when the generated high-resolution image data is handed over to the reconstruction unit 14), the decomposed low-resolution image data for which high-resolution image data is to be generated next is handed over from the temporary storage unit 12 to the AI ​​model 20 for which processing has been completed. The specific configuration of the AI ​​model 20 will be described later.

[0027] The reconstruction unit 14 reconstructs the image using the high-resolution image data generated from the AI ​​model 20. Reconstructing the image means constructing a video to be played on the screen of the client terminal 6 while managing the playback position of the high-resolution image (frame) data.

[0028] The display unit 15 displays the reconstructed high-resolution image data on the screen of the client terminal 6, thereby allowing the user 5 to view the seminar in real time. The high-resolution image data may be displayed on the screen using the browser of the client terminal 6, or may be displayed on the screen using application software downloaded to the client terminal 6.

[0029] FIG. 2 is a diagram showing the structure of FSRCNN, which is an example of an AI model according to an embodiment of the present invention.

[0030] Deep learning algorithms that utilize high-resolution technology include, for example, linear models, residual networks (ResNets), multi-branch models, recurrent models, incremental models, attention-based models, and GAN (generative adversarial networks) models.

[0031] Linear models are easy to use because they do not have a complex structure and do not cause communication connection interruptions, and they are sometimes adopted for smartphone devices due to their suitability in terms of performance.

[0032] There are two types of linear models for increasing the spatial frequency of an image: pre-upsampling and post-upsampling. Pre-upsampling extracts samples from a low-resolution image to a size suitable for high resolution, and then increases the resolution by learning hierarchical features. However, as the process progresses, the convolutional layers are layered, which requires time for calculations. Therefore, post-upsampling is used to learn the data of the low-resolution image and characterize it with an approximate linear model.

[0033] FSRCNN (Fast Super-Resolution Convolutional Neural Network) is a linear model with a posteriori upsampling.

[0034] FSRCNN is a type of convolutional neural network that is an improved model aimed at speeding up SRCNN (Super-Resolution Convolutional Neural Network), a method that uses CNN (Convolutional Neural Network) for super-resolution.

[0035] In this specification, FSRCNN will be used as an example.

[0036] FSRCNN consists of four consecutive convolutional layers and one deconvolutional layer. The four convolutional layers perform feature extraction, reduction, nonlinear transformation, and expansion, while the deconvolutional layer performs deconvolution.

[0037] In the first training phase of the FSRCNN, training was performed using 91 publicly available image data. Then, in the second training phase, additional training was performed using 100 uncompressed bitmap image data. The 100 bitmap image data were high quality and sharp, but had low image complexity. Image complexity refers to the lower image complexity of a white image (RGB(255, 255, 255)) compared to a landscape image, for example. These training phases were performed using a data augmentation method to increase the training data by 19 times, then reducing it to 90%, 80%, and 70% of its original size. Images were also rotated by 90°, 180°, and 270°.

[0038] As a result of these learning steps, as shown in Figure 3, it was found that after the second stage of learning, higher resolution images could be generated compared to after the first stage of learning.

[0039] Returning to FIG. 2, in the four successive convolutional layers, low-resolution image data 301 is used as input data and feature extraction, reduction, nonlinear transformation, and expansion are performed.

[0040] The first convolutional layer 101 extracts features using 5-pixel by 5-pixel convolution processing. It directly learns the mapping from the input low-resolution image data 301 to the high-resolution image data 302 without using interpolation.

[0041] The correction information is information that indicates how the input low-resolution image data 301 should be reproduced as high-resolution image data 302, and is generated from the input low-resolution image data 301 in the first convolutional layer 101.

[0042] The second convolution layer 102 performs 1 pixel x 1 pixel convolution processing to reduce the feature amount d of the low-resolution image data 301 to s, where d>>s.

[0043] The third convolutional layer 103 sets the number of mapping layers m and performs multiple 3-pixel x 3-pixel convolution processes. For example, two layers of 3-pixel x 3-pixel convolution processes cover the area of ​​5-pixel x 5-pixel convolution processes. Furthermore, the number of parameters for one layer of 5-pixel x 5-pixels is 25 (= 5 x 5), while the number of parameters for two layers of 3-pixel x 3-pixels is 18 (= 3 x 3 + 3 x 3), resulting in a 28% reduction in the number of parameters. Furthermore, learning with fewer parameters allows for faster convergence and reduces the possibility of overfitting. The number of mapping layers m has the greatest impact on the accuracy of the model.

[0044] The variables d, s, and m are described below.

[0045] The fourth convolutional layer 104 performs the opposite process to the second convolutional layer 102, increasing the feature value from s to d through 1 pixel x 1 pixel convolution processing, thereby expanding the feature value of the high-resolution image data 302.

[0046] The final layer, the deconvolutional layer 201, uses a 9 pixel x 9 pixel filter to reproduce high-resolution image data 302 while referencing the correction information generated in the first convolutional layer 101.

[0047] In FSRCNN, low-resolution image data, including correction information, is directly input to the convolutional layers 101 to 104, and then resolution enhancement is performed in the final layer, the deconvolutional layer 201, so that only a small amount of correction information can be used in the convolutional layers. By achieving low resolution and reduced correction information, FSRCNN achieves a lighter computational load and less complex memory.

[0048] The overall structure of FSRCNN can be expressed as FSRCNN(d, s, m), and its computational complexity can be expressed by the following formula: JPEG0007752353000001.jpg6170

[0049] The low-resolution images for the loss function are created by reducing the high-resolution image data used as training data. During the training process, the reduced high-resolution image data is used as the original data before training. The high-resolution image data generated during the training process is compared with the original data using mean squared error (MSE) to verify the degree to which it has been restored. The mean squared error can be calculated using the following formula. JPEG0007752353000002.jpg13170

[0050] JPEG0007752353000003.jpg5170 and JPEG0007752353000004.jpg4170 is a combination of the i-th low-resolution image data and high-resolution image data from the training data, JPEG0007752353000005.jpg6170 has a parameter θ This is the result of JPEG0007752353000006.jpg5170. In the learning process, Adam (Adaptive Moment Estimation), an algorithm for optimizing neural networks, is used.

[0051] In FSRCNN, image resolution is increased at the final layer, eliminating the need for the interpolation process previously required. As a result, the FSRCNN model has high learning accuracy for generating high-resolution image data from low-resolution image data, and because it can reduce the input image size, it can also reduce the filter size for feature extraction. This not only reduces processing complexity and memory usage, but also achieves more efficient and faster processing.

[0052] Figure 4 is a table summarizing the verification results for each model of FSRCNN.

[0053] The higher the LPIPS (Learned Perceptual Image Patch Similarity) value, the greater the difference from the original image, so a large value indicates a low image recall. On the other hand, a small value indicates a small difference from the original image, so a small value indicates a high image recall.

[0054] When the number of parameters in the FSRCNN model decreases, the image quality decreases (LPIPS value increases) and the processing speed increases. When the number of parameters in the FSRCNN model increases, the image quality increases (LPIPS value decreases) and the processing speed decreases.

[0055] The data used for the verification was a sample dataset called "Set5," which is commonly used when verifying the accuracy of high-resolution image data.

[0056] Model 56 (FSRCNN(56,12,4)) is the prototype of FSRCNN, but because it has 12,464 parameters, when trying to perform real-time processing on a smartphone or other device, the processing speed cannot keep up due to limitations in CPU, memory, storage size, etc., making it unsuitable for real-time viewing.

[0057] To enable real-time viewing on smartphones and other devices, it is necessary to reduce the values ​​of the parameters d, s, and m. To reproduce high-quality images, the difference between the values ​​of d and s must be increased, and the value of m must also be reduced in line with the values ​​of d and s in order to maintain the characteristics of low-resolution image data.

[0058] Figure 4 shows the results of testing the image quality and processing speed of each FSRCNN model with adjusted parameters. The frame rate requirement for the testing was 10 frames per second or higher. Models that could meet this requirement and reproduce good image quality were Model 10 (FSRCNN(10, 3, 1)), Model 5 (FSRCNN(5, 3, 1)), and Model 3 (FSRCNN(3, 3, 1)), with Model 18 (FSRCNN(18, 3, 1)) being able to reproduce the best image quality. In particular, Model 18 (FSRCNN(18, 3, 1)) had such high image recall that its LPIPS value was the same as that of Model 32 (FSRCNN(32, 5, 1)).

[0059] The low-resolution image data is decomposed according to the frame rate (fps) listed in the table in Figure 4, depending on the FSRCNN model. For example, model 18 (FSRCNN(18, 3, 1)) decomposes the low-resolution image data at 12 fps.

[0060] Figure 5 compares the filter size, number of filters, number of channels, input images, and number of parameters for the four convolutional layers and one deconvolutional layer in Model 56 (FSRCNN(56,12,4)) and Model 18 (FSRCNN(18,3,1)) of the FSRCNN models. Note that Model 56 (FSRCNN(56,12,4)) repeats the convolution process four times in the convolutional layer 103, so the convolutional layer 103 in Figure 5 is labeled as 4Conv.

[0061] The four convolutional layers that make up FSRCNN can be expressed as Conv(f, n, c), and the single deconvolutional layer can be expressed as Deconv(f, n, c). The variables (f, n, c) used in the convolutional and deconvolutional layers indicate the filter size, number of filters, and number of channels, respectively.

[0062] Compared to model 56 (FSRCNN(56,12,4)), model 18 (FSRCNN(18,3,1)) has a smaller filter size, number of filters, and number of channels in all convolutional and deconvolutional layers, making it lighter.

[0063] When reproducing high-resolution images, the processing time increases, while shortening the processing time results in degradation of the image quality. However, model 18 (FSRCNN(18, 3, 1)) can reproduce better image quality without sacrificing processing speed.

[0064] In addition, the deconvolution layer of model 18 (FSRCNN(18, 3, 1)) has a Deconv(9, 1, 18) structure, and the stride (the interval at which deconvolution is applied) is set so that when low-resolution image data is input, high-resolution image data with twice the resolution of the low-resolution image data is generated.

[0065] 6 is a flowchart showing the processing procedure in the video distribution system 1. The processing procedure of this embodiment will be described below based on this flowchart.

[0066] First, the acquisition unit 10 in the video distribution system acquires low-resolution image data transmitted from the server 4 (step S1).

[0067] The decomposing unit 11 uses the low-resolution image data received from the server 4 to decompose the low-resolution image data into frames (step S2).

[0068] The temporary storage unit 12 temporarily stores the data of the low-resolution image decomposed into frames (step S3).

[0069] The processing unit 13 passes the low-resolution image data decomposed into each frame to the AI ​​model 20, which generates high-resolution image data (step S4).

[0070] The reconstructing unit 14 reconstructs the data of the high-resolution image using the data of the high-resolution image generated from the AI ​​model 20 (step S5).

[0071] The display unit 15 displays the data of the reconstructed high-resolution image on the screen of the client terminal 6 (step S6).

[0072] The presence or absence of high-resolution image data to be reconstructed is checked (check whether acquisition of low-resolution image data has been completed), and if it is determined that there is no high-resolution image data to be reconstructed (step S7: No), the process ends. If it is determined that there is high-resolution image data to be reconstructed (step S7: Yes), the process returns to step S1, low-resolution image data is acquired, and the process from step S1 to step S7 is repeated until it is determined that there is no high-resolution image data to be reconstructed.

[0073] The above-described embodiment of the present invention is a lightweight model 18 (FSRCNN(18,3,1)) that can be implemented on smartphone terminals, etc., and by arranging multiple AI models in parallel and performing processing, it is possible to speed up the processing for generating high-resolution image data, thereby ensuring real-time performance.

[0074] In the above embodiment, we have described a case where a seminar held online can be viewed in real time, but it is also possible to allow users to view events such as concerts and sporting events in real time as video through a video distribution system.

[0075] Furthermore, through a video distribution system, for example, one or more users may be allowed to view in real time as video images the situation in which they are playing an online game, a consumer game, or an arcade game.

[0076] Furthermore, the contents of an online salon (an online community using SNS (Social Networking Service) or the like) may be displayed as a video and be viewed by users in real time.

[0077] The above-described embodiments are merely examples of the present invention, and it goes without saying that the present invention is not limited to the above-described embodiments. [Explanation of symbols]

[0078] 1. Distribution System 3...electronic computer 4. Server 6. Client terminal 10...Acquisition part 11...Disassembly section 12...Temporary storage section 13 Processing section 14... Reconstruction section 15...Display section 20. AI Model 101, 102, 103, 104...convolutional layers 201... Deconvolutional Layer 301 Low-resolution image data 302 High-resolution image data

Claims

1. A terminal that receives low-resolution image data from a server, decomposition means for decomposing the low-resolution image data into frames; a processing means for arranging a plurality of FSRCNNs (Fast Super-Resolution Convolutional Neural Networks) with predetermined parameters (d, 3, 1) (d=18, 10, 5 or 3) in parallel, processing a plurality of frames of low-resolution images simultaneously, and generating high-resolution image data for each frame; a reconstruction means for reconstructing data of a plurality of high resolution images according to a reproduction position; a display means for displaying the reconstructed high-resolution image data on a screen; A terminal comprising:

2. A program that causes a computer installed in a device that receives low-resolution image data from a server to function, decomposition means for decomposing the low-resolution image data into frames; a processing means for arranging multiple FSRCNNs with predetermined parameters (d, 3, 1) (d=18, 10, 5, or 3) in parallel, processing multiple frames of low-resolution images simultaneously, and generating high-resolution image data for each frame; a reconstruction means for reconstructing data of a plurality of high resolution images according to a reproduction position; a display means for displaying the reconstructed high-resolution image data on a screen; A program that causes a computer to function.

3. An information processing method for a device that receives low-resolution image data from a server, comprising: The device is a first step of decomposing the low resolution image data frame by frame; A second step of arranging multiple FSRCNNs with predetermined parameters (d, 3, 1) (d=18, 10, 5, or 3) in parallel to simultaneously process multiple frames of low-resolution images and generate high-resolution image data for each frame; a third step of reconstructing the data of the plurality of high resolution images according to the reproduction position; a fourth step of displaying the reconstructed high-resolution image data on a screen; An information processing method comprising:

Images