Learning device, image processing device, transmitting / receiving device, and transmitting / receiving system

Abstract

To perform a learning of a down-scaling and an up-scaling by using a high resolution video in an end-to-end.SOLUTION: A learning device comprises: a down-scaling part that includes a learning function, and generates a low resolution video image by performing a down-scaling of an input learning video image; a neural network video image coding part that codes the low resolution video image to convert to a bit stream; a neural network video image decoding part that generates a low resolution coding and decoding video image by decoding the bit stream; and an up-scaling part that includes the learning function, and outputs a high resolution coding and decoding video image by performing the up-scaling of the low resolution coding and decoding video image. In the learning device, at least one machine learning of the down-scaling part and the up-scaling part is characterized so that the output high solution coding and decoding video image is matched to the learning video image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a learning device, a video processing device, a transmission / reception device, and a transmission / reception system, and particularly to a learning device, a video processing device, a transmission / reception device, and a transmission / reception system that perform video downscaling or upscaling.

Background Art

[0002] When transmitting high-resolution videos such as 4K videos and 8K videos over a transmission path with a small available bandwidth, video downscaling is performed on the transmission side and then compressed by a video coding method such as the HEVC (High Efficiency Video Coding) method or the VVC (Versatile Video Coding) method and transmitted, and upscaling is performed on the reception side, it is known that the coding efficiency is improved (Non-Patent Document 1).

[0003] FIG. 11 is a conceptual diagram of a transmission / reception system that downscales and transmits an 8K video. The transmission / reception system of FIG. 11 includes a downscaler 1, an encoder 2, a decoder 3, and an upscaler 4, an 8K video is input on the transmission side, and an 8K reconstructed video is output on the reception side.

[0004] The downscaler 1 downscales the input 8K video by 1 / 2 to generate a 4K video and outputs it to the encoder 2. In prior research (Non-Patent Document 1), the bicubic method is used for downscaling on the transmission side.

[0005] The encoder 2 encodes the downscaled 4K video. As the coding method, general video coding technologies such as the HEVC method or the VVC method can be used. The encoder 2 converts the 4K video into a bitstream suitable for the bandwidth of the transmission path and transmits the bitstream.

[0006] Decoder 3 receives the bitstream and decodes it using a method corresponding to the encoding scheme of encoder 2. Here, decoder 3 generates a 4K decoded video and outputs it to upscaler 4.

[0007] Upscaler 4 upscales the input 4K decoded video by a factor of two to generate 8K video (reconstructed video). In previous research, the Lanczos method was used to upscale 4K video to 8K video at the receiving end. By combining this resolution conversion and encoding, encoding efficiency is improved compared to methods that encode and transmit 8K video without resolution conversion. In other words, image quality can be improved with the same bandwidth, and the bandwidth required for transmission can be reduced with the same image quality.

[0008] Furthermore, previous studies have shown that using a neural network-based super-resolution technique instead of the Lanczos method improves encoding efficiency compared to the Lanczos method. In these previous studies, a neural network model called SRFBN (Super-Resolution with Feedback Network: Non-Patent Literature 2) was used for upscaling as the super-resolution technique. Generally, when training a neural network model for super-resolution, the input is a low-resolution image obtained by downscaling the original high-resolution image, and the output is the original high-resolution image. However, in order to train the neural network model on the encoding degradation characteristics of HEVC and VVC methods, the input may be a low-resolution decoded image obtained by downscaling the original high-resolution image and then encoding and decoding it, and the output (ground truth data) may be the original high-resolution image. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Charles Bonnineau, Wassim Hamidouche, Jean-Francois Travers and Olivier Deforges, “Versatile Video Coding and Super-Resolution for Efficient Delivery of 8K Video with 4K Backward-Compatibility”, In ICASSP 2020, pp.2048-2052. [Non-Patent Document 2] Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu, “Feedback network for image Super-Resolution”, In CVPR 2019. (arXiv:1903.09814) [Non-Patent Document 3] Heewon Kim, Myungsub Choi, Bee Lim, and Kyoung Mu Lee, “Task-Aware Image Downscaling”, In ECCV 2018. [Non-Patent Document 4] Mingqing Xiao, Shuxin Zheng, Chang Liu, Yaolong Wang, Di He, Guolin Ke, Jiang Bian, Zhouchen Lin, and Tie-Yan Liu, “Invertible Image Rescaling”, In ECCV 2020. (arXiv:2005.05650) [Non-Patent Document 5] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai and Zhiyong Gao, “DVC: An End-to-end Deep Video Compression Framework”, In CVPR 2019. (arXiv:1812.00101) [Summary of the Invention] [Problems that the invention aims to solve]

[0010] Previous research has improved coding efficiency by using neural network-based super-resolution techniques for upscaling on the receiving end, but has used bicubic methods for downscaling on the transmitting end, and has not attempted to improve coding efficiency using neural networks for downscaling on the transmitting end.

[0011] In training a neural network that performs upscaling on the receiving end, it is possible to perform a basic level of training by using a low-resolution image obtained by downscaling and then encoding / decoding a high-resolution original image as the input, and using the high-resolution original image as the output (ground truth data). On the other hand, when training a neural network that performs downscaling on the transmitting end, it is necessary to perform end-to-end training not only through downscaling but also through encoding and decoding. However, since the backpropagation method, a neural network training method, cannot be used with video encoding methods such as HEVC and VVC, it is not possible to train a neural network that performs downscaling on the transmitting end.

[0012] Therefore, in view of the above-mentioned problems, the object of the present invention is to provide a learning device that can perform downscaling and upscaling learning end-to-end, including encoding and decoding, and to provide a video processing device, a transmitting / receiving device, and a transmitting / receiving system that can improve encoding efficiency compared to conventional resolution conversion, even when compressing and transmitting video using video encoding methods such as HEVC and VVC.

[0013] In this specification, high-resolution video refers to video before downscaling, which has a higher resolution than the video after downscaling. Low-resolution video refers to video after downscaling or before upscaling, which has a lower resolution than the video before downscaling or after upscaling. [Means for solving the problem]

[0014] To solve the above problems, the learning device according to the present invention comprises a downscaling unit having a learning function that downscales an input learning video to generate a low-resolution video; a neural network video encoding unit that encodes the low-resolution video and converts it into a bitstream; a neural network video decoding unit that decodes the bitstream to generate a low-resolution encoded and decoded video; and an upscaling unit having a learning function that upscales the low-resolution encoded and decoded video to output a high-resolution encoded and decoded video, wherein machine learning is performed on at least one of the downscaling unit and the upscaling unit so that the output high-resolution encoded and decoded video matches the learning video.

[0015] Furthermore, it is desirable that the learning device consists of a neural network comprising both the downscaling unit and the upscaling unit.

[0016] To solve the above problems, the learning device according to the present invention has a learning function and comprises a downscaling unit that downscales an input learning video to generate a low-resolution video, a neural network video encoding unit that encodes the low-resolution video and converts it into a bitstream, and a neural network video decoding unit that decodes the bitstream and outputs a low-resolution encoded and decoded video, wherein at least the downscaling unit performs machine learning so that the output low-resolution encoded and decoded video matches the low-resolution video generated by downscaling the learning video using predetermined arithmetic processing.

[0017] To solve the above problems, the present invention provides an image processing device that performs image reduction, comprising a downscaling unit equipped with a trained model that downscales an input video to generate a low-resolution video, wherein the trained model is a trained model of the downscaling unit in which the downscaling unit, encoding unit, decoding unit, and upscaling unit are all composed of neural networks, and machine learning has been performed so that the output video from the upscaling unit after a series of processes matches the input video to the downscaling unit.

[0018] To solve the above problems, the present invention provides an image processing device that performs image reduction, comprising a downscaling unit equipped with a trained model that downscales an input video to generate a low-resolution video, wherein the trained model is a trained model of the downscaling unit in which the downscaling unit, encoding unit, and decoding unit are all composed of a neural network, and machine learning has been performed so that the output video from the decoding unit after a series of processing of the input video matches the video generated by downscaling the input video using predetermined arithmetic processing.

[0019] To solve the above problems, the transmission device according to the present invention is characterized by comprising a video processing device that downscales an input video, and a video encoding unit that encodes the output video from the video processing device using a predetermined encoding method and converts it into a bitstream.

[0020] Furthermore, it is desirable that the transmission device uses either the HEVC or VVC encoding scheme as its predetermined encoding method.

[0021] In order to solve the above problems, a video processing apparatus according to the present invention has an upscaling unit including a learned model that upscales an input video to generate a high-resolution video, and is a video processing apparatus that performs image enlargement. The learned model is configured entirely of neural networks for a downscaling unit, an encoding unit, a decoding unit, and an upscaling unit, and machine learning is performed so that the output video from the upscaling unit after a series of processes matches the input video to the downscaling unit. It is a learned model of the upscaling unit.

[0022] In order to solve the above problems, a receiving apparatus according to the present invention includes a video decoding unit that decodes an input bit stream by a predetermined encoding method to generate an encoded / decoded video, and the video processing apparatus that upscales the encoded / decoded video.

[0023] In order to solve the above problems, a transmission / reception system according to the present invention includes the transmission apparatus and the receiving apparatus.

Advantages of the Invention

[0024] According to the learning apparatus in the present invention, learning of downscaling and upscaling can be performed end-to-end including encoding and decoding. Further, according to the video processing apparatus, transmission / reception apparatus, and transmission / reception system of the present invention, even when video is compressed and transmitted by a video encoding method such as the HEVC method or the VVC method, the encoding efficiency can be improved as compared with conventional resolution conversion.

Brief Description of the Drawings

[0025] [Figure 1] It is an example of a block diagram of a learning apparatus according to a first embodiment of the present invention. [Figure 2] It is a flowchart showing an example of a learning algorithm of a learning apparatus according to the first embodiment. [Figure 3] It is an example of a block diagram of a system that performs inference using a learned model. [Figure 4] This is an example of a block diagram of a learning device, a modified example of the first embodiment. [Figure 5] This flowchart shows an example of a learning algorithm for a modified learning device of the first embodiment. [Figure 6] This is an example of a block diagram of a video processing device according to a second embodiment of the present invention. [Figure 7] This is an example of a block diagram of a transmitting device according to a third embodiment of the present invention. [Figure 8] This is an example of a block diagram of a video processing device according to a fourth embodiment of the present invention. [Figure 9] This is an example of a block diagram of a receiving device according to a fifth embodiment of the present invention. [Figure 10] This is an RD curve calculated from the encoding results of the present invention and the conventional method. [Figure 11] This is a conceptual diagram of a transmission and reception system that downscales and transmits 8K video. [Modes for carrying out the invention]

[0026] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0027] (First embodiment) A learning device according to the first embodiment of the present invention learns a system that transmits high-resolution video by performing downscaling, encoding, decoding, and upscaling.

[0028] Figure 1 is an example of a block diagram of a learning device according to the first embodiment of the present invention. The learning device 100 includes a frame acquisition unit 10, a downscaling unit 21, a neural network video encoding unit 22, a neural network video decoding unit 23, an upscaling unit 24, and an error determination unit 30. Of these, the downscaling unit 21, the neural network video encoding unit 22, the neural network video decoding unit 23, and the upscaling unit 24 together constitute a learning block 20.

[0029] The input to the learning device 100 is a high-resolution video, such as 4K or 8K video, which serves as the source image and is used as the learning video. Here, the video may be a still image. In this invention, the term "image" includes moving images (video). The output of the learning device 100 is a trained model (for example, the parameters of a neural network), which outputs the optimal parameters for the learning block 20 (particularly the downscaling unit 21 and the upscaling unit 24).

[0030] This learning device 100 can be configured as a whole using a single computer and program. Alternatively, each block may be configured with an independent computer and program, if necessary. The following describes each configuration.

[0031] The frame acquisition unit 10 acquires the input training video and temporarily holds it. The frame acquisition unit 10 sequentially outputs the frames of the held video to the downscaling unit 21. It also outputs the frames of the held video as training data (correct answer data) to the error determination unit 30.

[0032] The downscaling unit 21 has a learning function and is composed of, for example, a neural network. The downscaling unit 21 downscales the high-resolution video, which is the original input image, to a low-resolution video and outputs it to the neural network video encoding unit 22. In this embodiment, for example, 8K video is downscaled to 4K video, i.e., to half the resolution, but downscaling to any resolution such as 3 / 4 times or 1 / 4 times may also be performed.

[0033] The neural network model of the downscaling unit 21 can be any model as long as it is a neural network that takes high-resolution video as input and low-resolution video as output. However, in this embodiment, either TAD (Task-Aware Downscaling: Non-Patent Literature 3), a resolution conversion method using a neural network, or IRN (Invertible Rescaling Network: Non-Patent Literature 4), a resolution conversion method that combines wavelet transform and a neural network, is used.

[0034] The neural network video encoding unit 22 encodes the input video (in this embodiment, the low-resolution video output of the downscaling unit 21) and converts it into a bitstream, and outputs the generated bitstream to the neural network video decoding unit 23.

[0035] The neural network video encoding unit 22 may use any video encoding scheme composed of a neural network. However, to improve video encoding efficiency when using video encoding schemes such as HEVC or VVC during inference, it is desirable to use a neural network model with an architecture similar to HEVC or VVC, such as the DVC (Deep Video Compression: Non-Patent Document 5) model. The DVC model, like HEVC and VVC, performs motion prediction and motion compensation on a block-by-block basis, and consistently performs transformation, quantization, and entropy encoding using a neural network. The neural network video encoding unit 22 compresses and encodes the video so that it becomes a bitstream corresponding to the set bandwidth.

[0036] The neural network video decoding unit 23 decodes the bitstream encoded by the neural network video encoding unit 22 and outputs the decoded video (sometimes called encoded-decoded video) to the upscaling unit 24. In this embodiment, a low-resolution encoded-decoded video is generated. The neural network video decoding unit 23 uses a video encoding scheme composed of a neural network, but it also uses the same video encoding scheme (encoding model) used in the neural network video encoding unit 22 for decoding.

[0037] The upscaling unit 24 has a learning function and is composed of, for example, a neural network. The upscaling unit 24 upscales the low-resolution encoded and decoded video, which is the output of the neural network video decoding unit 23, to a high-resolution encoded and decoded video. Depending on the downscaling performed by the downscaling unit 21, the upscaling is performed so that the low-resolution encoded and decoded video has the same resolution as the original high-resolution video. For example, if downscaling is performed to 1 / 2 the resolution, the upscaling is performed to 2 times the resolution; if downscaling is performed to 3 / 4 times the resolution, the upscaling is performed to 4 / 3 times the resolution.

[0038] The neural network model of the upscaling unit 24 can be any model as long as it is a neural network that takes low-resolution video as input and outputs high-resolution video. However, in this embodiment, we use TAU (Task-Aware Upscaling: Non-Patent Literature 3), which is a resolution conversion method using a neural network, or IRN, which is a resolution conversion method that combines wavelet transform and a neural network.

[0039] The error determination unit 30 uses the input video (high-resolution video) from the frame acquisition unit 10 as ground truth data to determine the error of the high-resolution encoded and decoded video output from the upscaling unit 24. The error determination unit 30 evaluates the prediction accuracy of the model using a general loss function and outputs the result to the learning block 20. The learning block 20 sequentially trains (modifies the parameters of) the models of the upscaling unit 24, neural network video decoding unit 23, neural network video encoding unit 22, and downscaling unit 21 using methods such as backpropagation to reduce the loss (error) calculated by the error determination unit 30.

[0040] Regarding the parameters to be learned, the parameters of all models in the downscaling unit 21, neural network video encoding unit 22, neural network video decoding unit 23, and upscaling unit 24 may be learned (modified), or the parameters of some models may be fixed and learned. For example, the parameters of the neural network video encoding unit 22 and the neural network video decoding unit 23 may be fixed, and the downscaling unit 21 and the upscaling unit 24 may be optimized. The learning device 100 performs machine learning on at least one of the downscaling unit 21 and the upscaling unit 24.

[0041] Furthermore, if only the downscaling unit 21 is trained, the upscaling unit may be constructed using a neural network with fixed parameters employing methods such as bicubic or Lanczos, considering cases where upscaling using a neural network cannot be performed on the receiving side due to computational complexity or hardware limitations.

[0042] After training using training videos is complete, the training device 100 outputs the trained models of the downscaling unit 21 and the upscaling unit 24, i.e., the optimal parameters of the neural network model.

[0043] In this embodiment, instead of using a video encoding method that relies on predetermined arithmetic processing such as the HEVC or VVC method, a video encoding method composed of a neural network is used. This enables end-to-end learning throughout the entire process of downscaling, encoding, decoding, and upscaling, that is, accurate learning using high-resolution video as the input and output of the series of processes. As a result, the neural network of the downscaling unit 21 on the transmitting side can be trained appropriately and accurately. Furthermore, the neural network of the upscaling unit 24 on the receiving side can be trained simultaneously with the downscaling unit 21 with greater accuracy. This learning method improves the overall encoding efficiency of the system compared to performing downscaling using calculations such as the bicubic method on the transmitting side.

[0044] Figure 2 shows a flowchart illustrating an example of the learning algorithm of the learning device 100 in this embodiment. Each step will be explained below.

[0045] Step S11: The learning device 100 acquires high-resolution video, which is used for learning. The frame acquisition unit 10 of the learning device 100 outputs the acquired high-resolution video to the downscaling unit 21.

[0046] Step S12: The downscaling unit 21 of the learning device 100 downscales the high-resolution video to a low-resolution video.

[0047] Step S13: The neural network video encoding unit 22 of the learning device 100 encodes the low-resolution video, converts it into a bitstream, and outputs it.

[0048] Step S14: The neural network video decoding unit 23 of the learning device 100 decodes the encoded bitstream and generates a low-resolution encoded and decoded video.

[0049] Step S15: The upscaling unit 24 of the learning device 100 upscales the generated low-resolution encoded and decoded video to a high-resolution encoded and decoded video.

[0050] Step S16: The error determination unit 30 of the learning device 100 uses the high-resolution video from the frame acquisition unit 10 as training data (correct data) to determine the error of the high-resolution encoded and decoded video, which is the output of the upscaling unit 24.

[0051] Step S17: Based on the error of the error determination unit 30, the learning device 100 learns the learning block 20 (downscaling unit 21, neural network video encoding unit 22, neural network video decoding unit 23, and upscaling unit 24) and corrects the parameters of each unit.

[0052] Step S18: The learning device 100 determines whether predetermined learning termination conditions (e.g., degree of error, number of learning iterations, etc.) are met. If the conditions are not met, it returns to the beginning (Step S11). If the conditions are met, it terminates the learning process and outputs the trained model (optimal parameters).

[0053] The learning of the learning block 20 by the learning device 100 of the present invention is performed based on the learning algorithm described above.

[0054] Figure 3 shows an example block diagram of a system that performs inference using a trained model. The system in Figure 3 uses a trained downscaling unit 21 and an upscaling unit 24 that were trained (or had their trained optimal parameters transferred) by the learning device 100 in Figure 1. However, for encoding and decoding during inference, a video encoding unit 25 and a video decoding unit 26, which are composed of any video encoding scheme such as HEVC or VVC, are used instead of a neural network. This is because the downscaling unit 21 and the upscaling unit 24 are intended to be used in a general transmission and reception system, that is, in a transmitting and receiving device that uses any video encoding scheme. By performing downscaling and upscaling using a trained neural network model, encoding efficiency is improved even when compressing and transmitting using a video encoding scheme such as HEVC or VVC during inference, compared to when resolution conversion is performed using conventional calculations such as the bicubic method.

[0055] In this system, the system switches between a video encoding scheme composed of a neural network and a video encoding scheme using predetermined calculations such as HEVC or VVC during training and inference. In the video encoding unit 25 of the inference system, encoding may be performed with an arbitrary quantization parameter QP, or QP may be selected to have a bitrate equivalent to that of the neural network encoding scheme used during training. As the prediction mode for encoding during inference, any mode from All Intra, Random Access, and Low Delay may be used, but since the DVC used in the training embodiment only uses forward-direction inter-screen prediction, it is desirable to use Random Access or Low Delay.

[0056] Let's briefly explain the inference process using the system shown in Figure 3. When high-resolution video is input, the downscaling unit 21 first downscales the high-resolution video to a low-resolution video based on the trained model.

[0057] Next, the video encoding unit 25 encodes the low-resolution video using a predetermined video encoding scheme (for example, HEVC or VVC) and generates a bitstream. The generated bitstream is output to the video decoding unit 26.

[0058] The video decoding unit 26 decodes the bitstream using an encoding scheme (for example, HEVC or VVC) corresponding to the video encoding unit 25, and generates a low-resolution encoded and decoded video.

[0059] The upscaling unit 24 upscales the low-resolution encoded / decoded video to a high-resolution encoded / decoded video based on the trained model. In this way, it performs inference using the trained model and outputs a high-resolution reconstructed video.

[0060] In the system shown in Figure 3, both the downscaling unit 21 and the upscaling unit 24 are pre-trained models, but a pre-trained model may be used for only one of them. For example, the upscaling unit 24 may use any upscaling method without using a neural network during inference. Even in this case, using a pre-trained model in the downscaling unit 21 improves the overall coding efficiency compared to the case where the bicubic method is used for downscaling on the transmitting side.

[0061] As described above, in the first embodiment, end-to-end learning was performed through downscaling, encoding, decoding, and upscaling, and the downscaling unit 21 and the upscaling unit 24 were learned simultaneously. Next, a modified version of the first embodiment will be described, which is a learning device that mainly learns the downscaling unit 21. In the modified version of the first embodiment, the upscaling unit 24 is omitted, making it possible to learn the downscaling unit 21 with a simpler configuration.

[0062] (Modified version of the first embodiment) A learning device, which is a modified version of the first embodiment of the present invention, learns a system for transmitting video by performing downscaling, encoding, and decoding.

[0063] Figure 4 shows an example of a block diagram of a modified learning device according to the first embodiment. The learning device 101 includes a frame acquisition unit 10, a downscaling unit 21, a neural network video encoding unit 22, a neural network video decoding unit 23, a low-resolution video generation unit 15, and an error determination unit 30. Of these, the downscaling unit 21, the neural network video encoding unit 22, and the neural network video decoding unit 23 together constitute a learning block 20.

[0064] The input to the learning device 101 is high-resolution video, such as 4K or 8K video, which serves as the source image and is used as the learning video. Here, the video may also be a still image. The output of the learning device 101 is a trained model (for example, the parameters of a neural network), and it outputs the optimal parameters for the learning block 20 (in particular, the downscaling unit 21).

[0065] This learning device 101 can be configured as a whole with a single computer and program. Alternatively, each block may be configured with an independent computer and program, if necessary. The following describes each configuration, but configurations common to the first embodiment shown in Figure 1 will be simplified in their explanation.

[0066] The frame acquisition unit 10 acquires the input training video and temporarily holds it. The frame acquisition unit 10 sequentially outputs the frames of the held video to the downscaling unit 21. It also outputs the frames of the held video to the low-resolution video generation unit 15.

[0067] The downscaling unit 21 has a learning function and is composed of, for example, a neural network. The neural network model of the downscaling unit 21 can be any model as long as it is a neural network that takes high-resolution video as input and low-resolution video as output, but as in the first embodiment, TAD or IRN is used. The downscaling unit 21 downscales the high-resolution video, which is the original input image, to a low-resolution video and outputs it to the neural network video encoding unit 22. For example, the downscaling unit 21 downscales 8K video to 4K video, that is, to half the resolution.

[0068] The neural network video encoding unit 22 encodes the input video (the low-resolution video output from the downscaling unit 21) and converts it into a bitstream, and outputs the generated bitstream to the neural network video decoding unit 23. The model of the neural network video encoding unit 22 may use a video encoding scheme composed of any neural network, but as in the first embodiment, it is desirable to use a neural network model with an architecture similar to the HEVC or VVC schemes, such as the DVC model. The neural network video encoding unit 22 compresses and encodes the video so that it becomes a bitstream corresponding to the set bandwidth.

[0069] The neural network video decoding unit 23 decodes the bitstream encoded by the neural network video encoding unit 22. In this modified example, a low-resolution encoded and decoded video is generated as the transmitted video and output to the error determination unit 30. The neural network video decoding unit 23 uses a video encoding scheme composed of a neural network, but it also uses the same video encoding scheme (encoding model) used in the neural network video encoding unit 22 for decoding.

[0070] The low-resolution video generation unit 15 generates a low-resolution video by downscaling using methods such as the bicubic method or the Lanczos method, i.e., by predetermined calculation processing. The low-resolution video generation unit 15 performs downscaling at the same magnification as the downscaling performed by the downscaling unit 21 (for example, to achieve a resolution of 1 / 2). The low-resolution video generation unit 15 outputs the generated low-resolution video to the error determination unit 30 as training data (ground truth data).

[0071] The error determination unit 30 uses the input video (low-resolution video) from the low-resolution video generation unit 15 as ground truth data to determine the error of the low-resolution encoded and decoded video, which is the output of the neural network video decoding unit 23. The error determination unit 30 evaluates the prediction accuracy of the model using a general loss function and outputs the result to the learning block 20. The learning block 20 sequentially trains (modifies the parameters of) the neural network video decoding unit 23, the neural network video encoding unit 22, and the downscaling unit 21 models using methods such as backpropagation to reduce the loss (error) calculated by the error determination unit 30.

[0072] Regarding the parameters to be learned, the parameters of all models in the downscaling unit 21, neural network video encoding unit 22, and neural network video decoding unit 23 may be learned (modified), or the parameters of some models may be fixed and learned. For example, the parameters of the neural network video encoding unit 22 and the neural network video decoding unit 23 may be fixed, and only the downscaling unit 21 may be optimized.

[0073] After training using training videos is complete, the training device 101 outputs the trained model of the downscaling unit 21, i.e., the optimal parameters of the neural network model.

[0074] In this modified version, instead of using video encoding methods that rely on predetermined computational processing, such as HEVC or VVC, a video encoding method composed of a neural network is used, enabling end-to-end learning throughout the downscaling, encoding, and decoding processes. This allows for proper and accurate training of the neural network in the downscaling unit 21 on the transmitting side. This learning method improves the overall encoding efficiency of the system compared to performing downscaling using computational methods such as the bicubic method on the transmitting side.

[0075] Figure 5 shows a flowchart illustrating an example of the learning algorithm for a modified learning device 101 of the first embodiment. Note that the flowchart in Figure 5 differs from the flowchart in Figure 2 in step S15'. Each step will be described below.

[0076] Step S11: The learning device 101 acquires high-resolution video, which is used for learning. The frame acquisition unit 10 of the learning device 101 outputs the acquired high-resolution video to the downscaling unit 21.

[0077] Step S12: The downscaling unit 21 of the learning device 101 downscales the high-resolution video to a low-resolution video.

[0078] Step S13: The neural network video encoding unit 22 of the learning device 101 encodes the low-resolution video, converts it into a bitstream, and outputs it.

[0079] Step S14: The neural network video decoding unit 23 of the learning device 101 decodes the encoded bitstream and generates a low-resolution encoded and decoded video.

[0080] Step S15': The low-resolution video generation unit 15 of the learning device 101 generates a low-resolution video from the high-resolution video from the frame acquisition unit 10 using a downscaling method that involves computational processing such as the bicubic method or the Lanczos method.

[0081] Step S16: The error determination unit 30 of the learning device 101 uses the low-resolution video from the low-resolution video generation unit 15 as training data (correct data) to determine the error of the low-resolution encoded and decoded video, which is the output of the neural network video decoding unit 23.

[0082] Step S17: The learning device 101 learns the learning block 20 (downscaling unit 21, neural network video encoding unit 22, and neural network video decoding unit 23) based on the error of the error determination unit 30, and modifies the parameters of each unit.

[0083] Step S18: The learning device 101 determines whether predetermined learning termination conditions (e.g., degree of error, number of learning iterations, etc.) are met. If the conditions are not met, it returns to the beginning (Step S11). If the conditions are met, it terminates the learning process and outputs the trained model (optimal parameters).

[0084] The learning of the learning block 20 by the learning device 101 in this modified example is performed based on the learning algorithm described above.

[0085] A trained downscaling unit 21, which has been trained using the learning device 101 (or has had its trained optimal parameters transferred), can be applied to any system that transmits video by performing downscaling, encoding, and decoding. That is, during inference, it is used in a video transmission system consisting of the trained downscaling unit 21 and a video encoding unit 25 and a video decoding unit 26, which are composed of any video encoding method such as HEVC or VVC, rather than a neural network. Furthermore, it can be used as the downscaling unit 21 of a system that performs inference using the trained model shown in Figure 3. By performing downscaling using the trained neural network model, even when compressing and transmitting using a video encoding method such as HEVC or VVC during inference, the encoding efficiency is improved compared to when resolution conversion by calculations such as the conventional bicubic method is used.

[0086] Figure 6 is an example of a block diagram of a video processing device according to a second embodiment of the present invention. The video processing device in this embodiment is an image reduction device 110. The image reduction device 110 includes a frame memory 11 and a downscaling unit 21. The image reduction device (video processing device) 110 downscales (reduces the image) an input video (high-resolution video) and outputs a low-resolution video (reduction image).

[0087] The frame memory 11 temporarily stores the input high-resolution video and sequentially sends the video frames to the downscaling unit 21. Note that the frame memory may be embedded within the downscaling unit 21 and is not required to be an independent block.

[0088] The downscaling unit 21 downscales the input video and outputs a low-resolution video with a reduced resolution. This downscaling unit 21 is equipped with a trained model that has been trained by the learning device 100 of the present invention shown in Figure 1. For example, the downscaling unit 21 is composed of a TAD or IRN model, which is a resolution conversion method using a neural network, and is created by transferring the optimal parameters (trained model) which are the output of the learning device 100.

[0089] Alternatively, the downscaling unit 21 may be a downscaling unit 21 equipped with a trained model that has been trained by the learning device 101 shown in Figure 4. For example, the downscaling unit 21 is composed of a TAD or IRN model, which is a resolution conversion method using a neural network, and is created by transferring the optimal parameters (trained model) which are the output of the learning device 101.

[0090] The image reduction device 110, which is an image processing device of this embodiment, can be applied to the transmitting side of a transmission and reception system that performs encoding and decoding. The trained model of the downscaling unit 21 of this embodiment is a trained model of the downscaling unit in which the downscaling unit, encoding unit, decoding unit, and upscaling unit are all composed of neural networks and have been trained end-to-end for a series of processes, that is, machine learning has been performed so that the output image from the upscaling unit matches the input image to the downscaling unit. Alternatively, the trained model of the downscaling unit 21 of this embodiment is a trained model of the downscaling unit in which the downscaling unit, encoding unit, and decoding unit are all composed of neural networks and have been trained end-to-end for a series of processes, that is, machine learning has been performed so that the output image from the decoding unit matches the image generated by downscaling the input image using predetermined arithmetic processing. By performing downscaling using this trained model, when compressing and transmitting using an image encoding method such as HEVC or VVC, the encoding efficiency is improved compared to conventional downscaling using calculations such as the bicubic method.

[0091] Figure 7 is an example of a block diagram of a transmitting device according to a third embodiment of the present invention. The transmitting device 120 comprises an image reduction device 110 and a video encoding unit 25. The transmitting device 120 receives high-resolution video as input and outputs a bitstream.

[0092] The image reduction device 110 is an image reduction device 110 of the second embodiment of the present invention, and has a downscaling unit 21 equipped with a trained model inside, which downscales the input high-resolution video and outputs a low-resolution video.

[0093] The video encoding unit 25 is configured with any video encoding method such as HEVC or VVC, encodes the input low-resolution video, and outputs a bitstream.

[0094] By using the transmitting device 120 of this embodiment as the transmitting device of a transmission / reception system, transmission and reception with improved encoding efficiency compared to conventional methods can be achieved.

[0095] Figure 8 is an example of a block diagram of a video processing device according to a fourth embodiment of the present invention. The video processing device in this embodiment is an image enlargement device 130. The image enlargement device 130 includes a frame memory 12 and an upscaling unit 24. The image enlargement device (video processing device) 130 upscales (enlarges) the input video (low-resolution video) and outputs a high-resolution video (enlarged image).

[0096] The frame memory 12 temporarily stores the input low-resolution video and sequentially sends the video frames to the upscaling unit 24. Note that the frame memory may be embedded within the upscaling unit 24 and is not required to be an independent block.

[0097] The upscaling unit 24 upscales the input video and outputs a high-resolution video with increased resolution. This upscaling unit 24 is equipped with a trained model that has been trained by the learning device 100 of the present invention shown in Figure 1. For example, the upscaling unit 24 is composed of a TAU or IRN model, which is a resolution conversion method using a neural network, and is created by transferring the optimal parameters (trained model) which are the output of the learning device 100.

[0098] The image processing device 130 of this embodiment, which is an image enlargement device, can be applied to the receiving side of a transmission and reception system that performs encoding and decoding. The trained model of the upscaling unit 24 of this embodiment is a trained model of the upscaling unit in which the downscaling unit, encoding unit, decoding unit, and upscaling unit are all composed of a neural network and have been trained end-to-end for a series of processes, that is, machine learning has been performed so that the output image from the upscaling unit matches the input image to the downscaling unit. By performing upscaling using this trained model, when receiving image that has been compressed and transmitted using an image encoding method such as HEVC or VVC, the encoding efficiency can be further improved compared to an upscaling unit that has been trained to correspond to conventional downscaling methods such as bicubic encoding.

[0099] Figure 9 is an example of a block diagram of a receiving device according to a fifth embodiment of the present invention. The receiving device 140 comprises a video decoding unit 26 and an image magnification device 130. The receiving device 140 receives a bitstream as input and outputs high-resolution video.

[0100] The video decoding unit 26 is configured with a video encoding scheme that corresponds to the transmitting side, such as HEVC or VVC, and decodes the input bitstream and outputs video (in this embodiment, low-resolution encoded and decoded video).

[0101] The image magnification device 130 is an image magnification device 130 of the fourth embodiment of the present invention, and has an upscaling unit 24 equipped with a trained model inside, which upscales the input low-resolution video and outputs a high-resolution video.

[0102] By using the receiving device 140 of this embodiment as a receiving device in a transmission / reception system, transmission and reception with improved coding efficiency compared to conventional methods can be achieved.

[0103] As a sixth embodiment of the present invention, a transmission and reception system can be constructed by combining the transmission device of the third embodiment and the reception device of the fifth embodiment. This transmission and reception system is equipped with a downscaling unit 21 and an upscaling unit 24 in the transmission device 120 and the reception device 140, which perform end-to-end learning of the series of processes, i.e., accurate learning using high-resolution video as input and output, thereby improving the overall coding efficiency of the transmission and reception system.

[0104] Generally, receivers used in terrestrial digital broadcasting have a function to perform pixel interpolation by upscaling, so this can be replaced, for example, by the image enlargement device of the present invention. The video processing device, transmitting and receiving device, and transmitting and receiving system of the present invention can be applied to terrestrial digital broadcasting to improve encoding efficiency.

[0105] (Experimental results and verification of effects) The experimental results of the present invention are presented below. The experiment was conducted under the conditions shown in Table 1. In the downscaling section, the present invention uses TAD as the neural network model and was compared with the conventional bicubic method. For the video encoding scheme, version 14.0 of VTM (VVC Test Model), a reference software for the VVC scheme, was used. Furthermore, assuming a case where upscaling by a neural network cannot be performed on the receiving side due to computational complexity or hardware limitations, both the present invention and the conventional method used the bicubic method in the upscaling section.

[0106] [Table 1]

[0107] The experiment was conducted using BQSquare, which is video from the JVET (Joint Video Experts Team) test sequence. BQSquare has a resolution of 432x240. In the downscaling section, it was reduced to a resolution of 216x120 (half the original resolution), then encoded and decoded using VTM, and finally enlarged to 432x240 (twice the original resolution) in the upscaling section.

[0108] Figure 10 shows the RD curve calculated from the encoding results of the present invention and the conventional method. The horizontal axis represents the bitrate, and the vertical axis represents the PSNR (Peak signal-to-noise ratio) of the luminance component when comparing the high-resolution original video with the high-resolution reconstructed video. The BD-Rate calculated from the encoding results was -51.89%, confirming that the present invention improves encoding efficiency.

[0109] In the first embodiment described above, the configuration and operation of the learning device 100 were explained, but the present invention is not limited thereto, and may be configured as a learning method for learning the downscaling unit and the upscaling unit. That is, it may be configured as a learning method comprising the steps of downscaling, encoding, decoding, and upscaling, based on the flowchart in Figure 2.

[0110] Furthermore, a computer can be suitably used to function as the learning devices 100, 101 or each of the blocks described above. Such a computer can be realized by storing a program in its memory that describes the processing content for realizing each function of the learning devices 100, 101 or each of the blocks, and by having the computer's CPU read and execute this program. This program can be recorded on a computer-readable recording medium.

[0111] Although the embodiments described above are representative examples, it will be apparent to those skilled in the art that many modifications and substitutions are possible within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the embodiments described above, and various modifications or changes are possible without departing from the scope of the claims. For example, the functions, etc., included in each block, step, etc., described in the embodiments can be rearranged in a logically consistent manner, and multiple constituent blocks, steps, etc., can be combined into one or divided. [Explanation of Symbols]

[0112] 1 Downscaler 2 encoders 3 Decoders 4 Upscalers 10 Frame acquisition section 11 Frame Memory 12 frame memory 15. Low-resolution video generation unit 20 Learning Blocks 21 Downscaling section 22 Neural Network Video Coding Unit 23 Neural Network Video Decoding Unit 24 Upscaling section 25 Video Encoding Section 26 Video Decoding Unit 30 Error judgment section 100 Learning Devices 110 Image reduction device 120 Transmitter 130 Image Magnification Device 140 Receiving device

Claims

1. A downscaling unit that has a learning function and generates a low-resolution image by downscaling the input learning video, A neural network video encoding unit that encodes the aforementioned low-resolution video and converts it into a bitstream, A neural network video decoding unit that decodes the bitstream and generates low-resolution encoded and decoded video, It includes an upscaling unit that has a learning function and upscales the low-resolution encoded / decoded video to output a high-resolution encoded / decoded video, A learning device characterized by performing machine learning on at least one of the downscaling unit and the upscaling unit so that the output high-resolution encoded and decoded video matches the training video.

2. In the learning device of claim 1, A learning device characterized in that the downscaling unit and the upscaling unit are composed of neural networks.

3. A downscaling unit that has a learning function and generates a low-resolution image by downscaling the input learning video, A neural network video encoding unit that encodes the aforementioned low-resolution video and converts it into a bitstream, The system includes a neural network video decoding unit that decodes the bitstream and outputs low-resolution encoded and decoded video. A learning device characterized in that at least the downscaling unit performs machine learning so that the output low-resolution encoded and decoded video matches a low-resolution video generated by downscaling the training video using predetermined arithmetic processing.

4. An image processing device that performs image reduction, having a downscaling unit equipped with a trained model that downscales input video to generate low-resolution video, The aforementioned trained model is a trained model for a downscaling unit, characterized in that the downscaling unit, the encoding unit that encodes the video and converts it into a bitstream, the decoding unit that decodes the bitstream and outputs the video, and the upscaling unit are all composed of neural networks, and machine learning has been performed so that the video output from the upscaling unit after a series of processes matches the video input to the downscaling unit.

5. An image processing device that performs image reduction, having a downscaling unit equipped with a trained model that downscales input video to generate low-resolution video, The aforementioned trained model is characterized in that the downscaling unit is a trained model of the downscaling unit, in which the downscaling unit, the encoding unit that encodes the video and converts it into a bitstream, and the decoding unit that decodes the bitstream and outputs the video are all composed of neural networks, and machine learning has been performed so that the output video from the decoding unit after a series of processing of the input video matches the video generated by downscaling the input video using predetermined arithmetic processing.

6. A video processing device according to claim 4 or 5 for downscaling an input video, A video encoding unit that encodes the video output from the video processing device using a predetermined encoding method and converts it into a bitstream. A transmitting device characterized by comprising:

7. In the transmitting device according to claim 6, A transmitting device characterized in that the predetermined encoding scheme is the HEVC scheme or the VVC scheme.

8. An image processing device that performs image enlargement, having an upscaling unit equipped with a trained model that upscales input video to generate high-resolution video, The aforementioned trained model is a trained model for an upscaling unit, characterized in that the downscaling unit, the encoding unit that encodes the video and converts it into a bitstream, the decoding unit that decodes the bitstream and outputs the video, and the upscaling unit are all composed of neural networks, and machine learning has been performed so that the video output from the upscaling unit after a series of processes matches the video input to the downscaling unit.

9. A video decoding unit that decodes the input bitstream using a predetermined encoding scheme and generates encoded and decoded video, The video processing apparatus according to claim 8, which upscales the encoded and decoded video, A receiving device characterized by comprising:

10. A transmitting and receiving system characterized by comprising a transmitting device according to claim 6 and a receiving device according to claim 9.

Images