Image processing device, upscaling method, and image processing system

Abstract

To provide a video processing device, an upscaling method, and a video processing system that improve encoding efficiency.SOLUTION: A video processing device 200 includes an upscaling unit 220 which inputs a first video signal of a first resolution video and a second video signal of a second resolution video having a higher resolution than the first resolution video, generates a third video signal of a third resolution video having a higher resolution than the second resolution video by upscaling the first video signal and the second video signal using a neural network, and outputs the third video signal.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an upscaling method, and an image processing system. [Background technology]

[0002] One method for encoding video signals is spatially scalable coding. Spatially scalable coding is supported in SHVC (Scalable HEVC) or VVC (Versatile Video Coding), which are extensions of HEVC (High-Efficiency Video Coding).

[0003] Spatial scalable coding is an encoding method that encodes multiple video signals with different spatial resolutions. In spatial scalable coding, each picture is layered into two layers: a base layer with a spatially lower resolution than the original image, and an enhancement layer with a spatially higher resolution than the base layer (or the same resolution as the original image). Then, in spatial scalable coding, encoding is performed on each video signal of the base layer image and the enhancement layer image. In spatial scalable coding, a predicted picture generated by upsampling the base layer is referenced when encoding the enhancement layer. This improves the encoding efficiency of the enhancement layer compared to cases where the predicted picture is not referenced.

[0004] Meanwhile, a technology called super-resolution (SR) is attracting attention. Super-resolution is a video processing technique that increases the resolution of input video and converts it into high-resolution video. Regarding super-resolution, there are super-resolution methods that use neural networks (for example, see Non-Patent Documents 1 and 2 below). Super-resolution methods using neural networks are known to have improved encoding efficiency compared to filtering-based super-resolution methods such as the bicubic method or the Lanczos method. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Charles Bonnineau, Wassim Hamidouche, Jean-Francois Travers and Olivier Deforges. “Versatile Video Coding and Super-Resolution for Efficient Delivery of 8k Video with 4k Backward-Compatibility”, In ICASSP 2020 (February 17, 2020). [Non-Patent Document 2] Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. Feedback network for image superresolution. In CVPR 2019 (June 28, 2019). [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] Therefore, this disclosure aims to provide an image processing device, an upscaling method, and an image processing system that improve encoding efficiency. [Means for solving the problem]

[0007] The video processing device according to the first embodiment has an upscaling unit that receives a first video signal of a first resolution video and a second video signal of a second resolution video with a higher resolution than the first resolution video, and generates a third video signal of a third resolution video with a higher resolution than the second resolution video by upscaling the first video signal and the second video signal using a neural network, and outputs the third video signal.

[0008] The upscaling method according to the second aspect inputs the first video signal of the first-resolution video and the second video signal of the second-resolution video with a higher resolution than the first-resolution video. Further, the upscaling method generates a third video signal of a third-resolution video with a higher resolution than the second-resolution video by performing upscaling on the first video signal and the second video signal using a neural network. Furthermore, the upscaling method outputs the third video signal.

[0009] The video processing system according to the third aspect is a video processing system having a video encoding device and a video decoding device. The video encoding device has a downscaling unit that generates a first video signal of a first-resolution video and a second video signal of a second-resolution video with a higher resolution than the first-resolution video by downscaling the third video signal of the third-resolution video. Further, the video encoding device has a video encoding unit that performs spatially scalable encoding on the first video signal and the second video signal and outputs the encoded first video signal and second video signal. On the other hand, the video decoding device has a video decoding unit that decodes the first video signal and the second video signal respectively by spatially scalable decoding of the encoded first video signal and second video signal. Further, the video decoding device has an upscaling unit that generates the third video signal related to the third-resolution video with a higher resolution than the second-resolution video by performing upscaling on the first video signal and the second video signal using a neural network, and outputs the third video signal.

Advantages of the Invention

[0010] According to the present disclosure, it is possible to provide a video processing device, an upscaling method, and a video processing system that improve encoding efficiency.

Brief Description of the Drawings

[0011] [Figure 1] FIG. 1 is a diagram showing a configuration example of a video processing system according to the first embodiment. [Figure 2] Figure 2 is a diagram showing a configuration example of the upscaling unit according to the first embodiment. [Figure 3] Figures 3(A) to 3(E) are diagrams for explaining each configuration of Figure 2. [Figure 4] Figure 4 is a diagram showing an operation example according to the first embodiment. [Figure 5] Figure 5 is a diagram showing a configuration example of the upscaling unit according to a modification of the first embodiment.

Mode for Carrying Out the Invention

[0012] [First Embodiment]

[0013] (Regarding Super-Resolution) As described above, super-resolution is, for example, a process of converting a low-resolution (LR) video into a high-resolution (HR) video.

[0014] For example, there may be a case of downscaling a high-resolution video into a low-resolution video. By transmitting the downscaled low-resolution video through a transmission path or recording it on a recording medium, transmission or recording can be performed with a smaller amount of data compared to the case of a high-resolution video.

[0015] However, even after receiving a low-resolution video or reading a low-resolution video from a recording medium and upscaling it to a high-resolution video, there may be cases where the high-resolution video cannot be accurately reproduced. This is because information lost due to downscaling the high-resolution video (especially information in the high-frequency region) may not be accurately reproduced.

[0016] Therefore, super-resolution techniques are sometimes used. In particular, super-resolution techniques using a neural network called SRFBN (Super-Resolution with Feedback Network) are known to improve the encoding efficiency of upscaled video compared to filtering-based methods, as described in Non-Patent Documents 1 and 2.

[0017] SRFBN takes low-resolution video as input and uses an end-to-end convolutional neural network (CNN) to output high-resolution video. In this process, SRFBN includes a feedback mechanism to iteratively minimize the loss between the reconstructed high-resolution video and the original high-resolution video before downscaling.

[0018] The super-resolution techniques using neural networks described in Non-Patent Documents 1 and 2, including SRFBN, are based on inputting a single image. Therefore, it is difficult to upscale multiple images using the neural network with these super-resolution techniques. Consequently, with these super-resolution techniques, inputting only a single image does not improve image quality compared to inputting multiple images, and encoding efficiency (for example, an indicator showing whether or not visual image quality improves even with the same amount of data) may not improve.

[0019] Here, we assume the following scenario: When transmitting low-resolution video and high-resolution video encoded using spatially scalable coding, there may be cases where sufficient transmission bandwidth cannot be provided. Therefore, at the transmitting end, instead of high-resolution video, an intermediate-resolution video with a lower resolution than the high-resolution video is used as the encoding target, and the low-resolution video and intermediate-resolution video are spatially scalable coded. Then, at the receiving end, a high-resolution video is obtained by using a super-resolution technique on the intermediate-resolution video decoded by spatially scalable decoding.

[0020] In this case as well, using the super-resolution technology using neural networks described in Non-Patent Documents 1 and 2, a single intermediate resolution video is input, and that intermediate resolution video is upscaled. Therefore, even in this case, the encoding efficiency may not be improved compared to when multiple videos are input.

[0021] Therefore, the objective of the first embodiment is to improve encoding efficiency. For example, in the first embodiment, even if intermediate resolution video is encoded by spatially scalable encoding and transmitted to the receiving side, the objective is to improve the encoding efficiency of the upscaled high-resolution video at the receiving side.

[0022] Embodiments will be described below with reference to the drawings. In the following drawings, identical or similar parts are denoted by the same or similar reference numerals.

[0023] (Example of a video processing system configuration) Figure 1 is a diagram showing an example configuration of the video processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the video processing system 10 includes a video encoding device 100 and a video decoding device 200. The video encoding device 100 also includes a downscaling unit 110 and a video encoding unit 120.

[0025] The downscaling unit 110 generates a low-resolution video signal (the first video signal of the first-resolution video) and an intermediate-resolution video signal (the second video signal of the second-resolution video) which has a higher resolution than the low-resolution video, by downscaling the high-resolution video signal (the third video signal of the third-resolution video).

[0026] The downscaling method may be a filtering-based method such as the bicubic method or the Lanczos method. The bicubic method is a method in which, for example, weighted pixel values ​​using a predetermined cubic equation are interpolated as the pixels to be processed for 4x4 pixels. The Lanczos method is a method in which, for example, weighted pixel values ​​using the sinc function are interpolated as the pixels to be processed for pixels within a predetermined range (2x2, 3x3, etc.). Any filtering-based downscaling method other than the bicubic method or the Lanczos method may be used.

[0027] Furthermore, the downscaling method may include a resolution transformation technique using a neural network. Examples of such neural network-based resolution transformation techniques include TAD (Task-Aware Downscaling) or IRN (Invertible Rescaling Network).

[0028] TAD is a resolution conversion method described in Non-Patent Document 3 (Heewon Kim, Myungsub Choi, Bee Lim, and Kyoung Mu Lee. Task-Aware Image Downscaling. In ECCV, 2018.). TAD includes a downscaling layer, three residual blocks, and a residual connection. The downscaling layer uses a pixel shuffle layer to reduce the resolution. Furthermore, the degradation of learning accuracy is suppressed by skipping the three residual blocks using residual connections. With TAD, a downsampled intermediate resolution output image can be obtained from a high-resolution input image.

[0029] Furthermore, IRN is a resolution conversion method described in Non-Patent Document 4 (Mingqing Xiao, Shuxin Zheng, Chang Liu, Yaolong Wang, Di He, Guolin Ke, Jiang Bian, Zhouchen Lin, and Tie-Yan Liu. Invertible image rescaling. In ECCV, 2020 (May 12, 2020).). IRN includes a Haar function and an invertible neural network block. The Haar function decomposes high-resolution video into low-frequency and high-frequency components. The invertible neural network block models the distribution of high-frequency information lost during downscaling, and an intermediate-resolution output video is obtained from the low-frequency and high-frequency components.

[0030] When a neural network-based downscaling method is used, the downscaling unit 110 may generate low-resolution and intermediate-resolution video using separate models for each resolution, or it may generate video of multiple resolutions from the same model.

[0031] The downscaling unit 110 uses this downscaling method to obtain a low-resolution video signal (e.g., 2K video) and an intermediate-resolution video signal (e.g., 4K video) from a high-resolution video signal (e.g., 8K video). The downscaling unit 110 may output multiple intermediate-resolution video signals (e.g., 4K video and 6K video) with respect to the intermediate-resolution video.

[0032] The video encoding unit 120 performs spatially scalable encoding on the video signal of the low-resolution video (base layer) and the video signal of the intermediate-resolution video (enhancement layer), and outputs two encoded video signals. The video encoding unit 120 is not limited to one intermediate-resolution video (e.g., 4K video), but may encode the video signals of multiple intermediate-resolution videos (e.g., 4K video and 6K video) as the enhancement layer. For spatially scalable encoding, known spatially scalable encoding methods used in SHVC or VVC may be used.

[0033] The video encoding unit 120 may also multiplex the encoded low-resolution video signal and the intermediate-resolution video signal to output the two encoded video signals as a single bitstream to the video decoding device 200.

[0034] The video decoding device 200 includes a video decoding unit 210 and an upscaling unit 220. The video decoding device 200 may also be a video processing device or an upscaling device.

[0035] The video decoding unit 210 decodes the low-resolution video signal and the intermediate-resolution video signal, which have been encoded by spatially scalable coding, back into the low-resolution video signal and the intermediate-resolution video signal, respectively, by spatially scalable decoding. The video decoding unit 210 may also receive the bitstream output from the video encoding device 100 and extract the encoded video signal from the bitstream. The video decoding unit 210 outputs the decoded low-resolution video signal and the intermediate-resolution video signal to the upscaling unit 220, as well as to other processing blocks inside or outside the video decoding device 200.

[0036] The upscaling unit 220 receives a video signal of a low-resolution video (the first video signal of the first resolution video) and a video signal of an intermediate resolution video with a higher resolution than the low-resolution video signal (the second video signal of the second resolution video). The upscaling unit 220 then uses a neural network to upscale the low-resolution video signal and the intermediate resolution video signal to generate a high-resolution video signal with a higher resolution than the intermediate resolution video (the third video signal of the third resolution video). The upscaling unit 220 is not limited to using just one intermediate resolution video; it may use multiple intermediate resolution videos (for example, 4K video and 6K video) to generate the high-resolution video signal.

[0037] The neural network used in the upscaling unit 220 can be any model as long as it can accept video signals from multiple resolutions. In the following explanation, we will use a model that extends the TAU (Task-Aware Upscaling) described in Non-Patent Document 3 to accept input from multiple resolutions.

[0038] Figure 2 is a diagram showing an example configuration of the upscaling unit 220 according to the first embodiment. Figures 3(A) to 3(E) are diagrams illustrating the neural network used in Figure 2.

[0039] In Figure 2, a low-resolution image is shown as one image I. l Therefore, the intermediate resolution video is two videos I m1 and I m2 This illustrates an example. For example, low-resolution video I l 2K video, intermediate resolution video I m1 4K video, intermediate resolution video I m2 The following explanation assumes that each of these is 6K video.

[0040] As shown in Figure 2, the upscaling unit 220 processes the low-resolution video I l In addition, it has a first upscaling module 221-1 and a first feature extraction module 222-1. Furthermore, the upscaling unit 220 has an intermediate resolution video Im2 On the other hand, it has a second upscaling module 221-2. Further, the upscaling unit 220 has an intermediate-resolution video I m1 On the other hand, it has a third upscaling module 221-3 and a third feature extraction module 222-3.

[0041] The first upscaling module 221-1 uses a neural network (first neural network) to upscale the video signal (first video signal) of the low-resolution video I l to the video signal of the intermediate-resolution video I m2 by upscaling.

[0042] First, the first upscaling module 221-1 includes three residual blocks between two convolutional layers and has a residual connection that skips the residual blocks. As shown in FIG. 3(E), the residual block has two convolutional layers and a ReLU function, and has a configuration in which the input of the residual block is added to the output of the residual block.

[0043] Second, the first upscaling module 221-1 includes a pixel shuffle layer. In the pixel shuffle layer, upscaling to the intermediate-resolution video I is performed on the feature amount extracted by the convolutional layer with respect to the output after the residual connection. m2 by upscaling.

[0044] The pixel shuffle layer is described in Non-Patent Document 5 (Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert and Zehan Wang. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In CVPR 2016.). The pixel shuffle layer is sometimes referred to as sub-pixel convolution or ESPCN (Efficient Sub-Pixel Convolutional Neural Network).

[0045] The pixel shuffle layer performs convolution on the input image to increase the number of pixels, and then rearranges the pixels according to the scaling factor to output an upscaled image. As mentioned above, convolution is performed by periodically activating a part of the filter according to the position of the subpixels. In the pixel shuffle layer, feature quantities are extracted by the convolution layer and then upscaled at the end, which allows for a smaller filter size compared to upscaling first, thus reducing memory usage and other factors.

[0046] The first feature extraction module 222-1 extracts upscaled low-resolution video I l The first feature extraction module 222-1 includes a convolutional layer and a residual block. The first feature extraction module 222-1 uses such a neural network to extract features from the low-resolution video I l The high-frequency information contained in the video signal is used in low-resolution video I l The features are extracted from the video signal. The first feature extraction module 222-1 extracts the low-resolution video Il The characteristic features of the video signal are output to the second upscaling module 221-2.

[0047] The third upscaling module 221-3 processes intermediate resolution video I m1 and I m2 (For example, 4K video and 6K video) The video with the highest resolution I m2 Video I other than (e.g., 6K video) m1 The video signal (fourth video signal), such as 4K video, is upscaled using a neural network (first neural network). The neural network model included in the third upscaling module 221-3 is identical to that of the first upscaling module 221-1. The third upscaling module 221-3 upscales intermediate resolution video I m1 The video signal is an intermediate resolution video I m2 The video signal is upscaled.

[0048] The third feature extraction module 222-3 extracts the upscaled intermediate resolution video I m1 Features are extracted from the video signal (fourth video signal). The neural network model included in the third feature extraction module 222-3 is identical to the model in the first feature extraction module 222-1. The third feature extraction module 222-3 extracts features from intermediate resolution video I m1 The high-frequency information contained in the video signal is used in intermediate resolution video I m1 The features are extracted from the video signal. The third feature extraction module 222-3 outputs the extracted features to the second upscaling module 221-2.

[0049] Note that the features output by the first feature extraction module 222-1 and the features output by the third feature extraction module 222-3 are, respectively, low-resolution video I l and intermediate resolution video I m1 Because they are based on different features, they have different characteristics.

[0050] The second upscaling module 221-2 upscales the video signal with the highest resolution (e.g., 6K video) among the video signals of intermediate resolution video (e.g., 4K video and 6K video). At that time, the second upscaling module 221-2 upscales the low-resolution video I l The feature quantities of the video signal (first video signal) and the upscaled intermediate resolution video I m1 Upscaling is performed using the feature quantities of the video signal (fourth video signal). Specifically, the second upscaling module 221-2 uses the intermediate resolution video I m2 The feature quantities for the video signal are added together with the two feature quantities output from the first feature extraction module 222-1 and the third feature extraction module 222-3. Then, the second upscaling module 221-2 uses a pixel shuffle layer to process the high-resolution video I h The video signal (third video signal) is upscaled. The second upscaling module 221-2 performs high-resolution video I h The video signal is output to processing blocks inside or outside the video decoding device 200.

[0051] (Example of operation according to the first embodiment) Figure 4 is a diagram illustrating an example of operation according to the first embodiment.

[0052] As shown in Figure 4, in step S10, the video processing system 10 starts processing.

[0053] In step S11, the downscaling unit 110 downscales the input high-resolution video signal (the third video signal of the third-resolution video) into a low-resolution video signal (the first video signal of the first-resolution video) and an intermediate-resolution video signal (the second video signal of the second-resolution video).

[0054] In step S12, the video encoding unit 120 encodes the video signal of the low-resolution video and the video signal of the intermediate-resolution video using spatially scalable encoding.

[0055] In step S13, the video decoding unit 210 decodes the encoded low-resolution video signal and the intermediate-resolution video signal by spatially scalable decoding.

[0056] In step S14, the upscaling unit 220 uses a neural network to upscale the low-resolution video signal and the intermediate-resolution video signal after decoding, thereby generating a high-resolution video signal.

[0057] In step S15, the video processing system 10 completes the series of processes.

[0058] As described above, in the first embodiment, the upscaling unit 220 receives a video signal of a low-resolution video and a video signal of an intermediate-resolution video, and performs upscaling on at least two video signals using a neural network. In this case, the upscaling unit 220 upscales the video signal of the intermediate-resolution video using not only high-frequency information lost during the downscaling and encoding process of the intermediate-resolution video, but also high-frequency information lost during the downscaling and encoding process of the low-resolution video. As a result, for example, in the upscaling unit 220 according to the first embodiment, it is expected that the video quality will be improved compared to the case where a single video signal of an intermediate-resolution video is upscaled, and thus it will be possible to improve the encoding efficiency.

[0059] (Modification of the first embodiment) In the first embodiment, an example was described in which there are two intermediate resolution videos (a 4K video and a 6K video), but the number of intermediate resolution videos input to the upscaling unit 220 is not limited to this, and for example, the number of intermediate resolution videos input may be "1".

[0060] Figure 5 shows an example configuration of the upscaling unit 220 when the number of intermediate resolution video inputs is "1". As shown in Figure 5, the upscaling unit 220 receives low resolution video I lFor the video signal, similar to the first embodiment, it has a first upscaling module 221-1 and a first feature extraction module 222-1. The first upscaling module 221-1 uses a neural network (first neural network) to extract low-resolution video I l The video signal (first video signal) is converted to intermediate resolution video I m The video signal is upscaled to the following resolution. The first feature extraction module 222-1 uses a neural network (second neural network) to upscale the low-resolution video I. l Features of the video signal are extracted. The neural network models included in the first upscaling module 221-1 and the first feature extraction module 222-1 are the same as those in the first embodiment.

[0061] Furthermore, the upscaling unit 220 processes intermediate resolution video I m For the video signal, a second upscaling module 221-2 is provided. The second upscaling module 221-2 uses a neural network (first neural network) to upscale the low-resolution video I l Using the feature quantities of the video signal, intermediate resolution video I m The video signal is upscaled. The neural network model included in the second upscaling module 221-2 is the same as in the first embodiment.

[0062] Note that the number of intermediate resolution video inputs to the upscaling unit 220 is not limited to "1" and "2," but may be "3" or more. In this case, the third upscaling module 221-3 and the third feature extraction module 222-3 are applied to the video signals of two or more intermediate resolution videos other than the one with the highest resolution. The third upscaling module 221-3 upscales the video signal to the highest resolution among the intermediate resolution videos. The third feature extraction module 222-3 extracts high-frequency information contained in the video signal. Then, the second upscaling module 221-2 upscales the video signal to high resolution using the high-frequency information of the low-resolution video, the high-frequency information contained in the video signal, and the high-frequency information of the highest-resolution intermediate resolution video.

[0063] [Other embodiments] The video processing system 10 may be provided as a single device as a video processing device. In this case, the video encoding device 100 may be a video encoding unit, and the video decoding device (or video processing device) 200 may be a video decoding unit (or video processing unit).

[0064] A program may be provided that causes a computer to execute each of the processes performed by the above-described device (video encoding device 100 and video decoding device 200). The program may be recorded on a computer-readable medium. Using a computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, but may be a recording medium such as a CD-ROM or DVD-ROM. Furthermore, the circuits that execute each of the processes performed by the above-described device (video encoding device 100 and video decoding device 200) may be integrated, and the device may be configured using a semiconductor integrated circuit (chipset, SoC).

[0065] Although the embodiments have been described in detail above with reference to the drawings, the specific configuration is not limited to those described above, and various design changes can be made without departing from the gist of the invention. Furthermore, it is possible to combine the various operational examples within a range that does not contradict each other. [Explanation of Symbols]

[0066] 100: Video encoding device 110: Downscaling unit 120: Video encoding unit 200: Video decoding device 210: Video decoding unit 220: Upscaling unit 221-1: First Upscaling Module 222-1: First Feature Extraction Module 221-2: Second Upscaling Module 221-3: Third Upscaling Module 222-3: Third Feature Extraction Module

Claims

1. The system includes an upscaling unit that receives a first video signal of a first resolution video and a second video signal of a second resolution video with a higher resolution than the first resolution video, and uses a neural network to upscale the first and second video signals to generate a third video signal of a third resolution video with a higher resolution than the second resolution video, and outputs the third video signal. The upscaling unit is, A first upscaling module that upscales the first video signal using the first neural network of the aforementioned neural network, and a first feature extraction module that extracts feature quantities from the upscaled first video signal using the second neural network of the aforementioned neural network, A second upscaling module generates a third video signal by upscaling the second video signal using the features of the first video signal with the first neural network, has Image processing device.

2. The upscaling unit is, When the second video signal includes a plurality of video signals with different resolutions, the system further comprises a third upscaling module that upscales the fourth video signal of the video signal other than the one with the highest resolution among the plurality of video signals using the first neural network, and a third feature extraction module that extracts feature quantities of the upscaled fourth video signal using the second neural network. The second upscaling module uses the upscaled features of the first video signal and the upscaled features of the fourth video signal to upscale the video signal of the video with the highest resolution among the plurality of video signals. The image processing apparatus according to claim 1.

3. The first video signal and the second video signal are video signals decoded by spatially scalable decoding. The image processing apparatus according to claim 1.

4. The first neural network includes residual blocks that skip the convolutional layers included in the first neural network. The second neural network includes residual blocks that skip the convolutional layers included in the second neural network. The image processing apparatus according to claim 1.

5. A first video signal of a first resolution video and a second video signal of a second resolution video with a higher resolution than the first resolution video are input. By upscaling the first and second video signals using a neural network, a third video signal with a higher resolution than the second resolution video is generated. The third video signal is output, The above generation is, Using the first neural network of the aforementioned neural network, the first video signal is upscaled; and using the second neural network of the aforementioned neural network, the feature quantities of the upscaled first video signal are extracted. The third video signal is generated by using the first neural network to upscale the second video signal using the features of the first video signal, including Upscaling methods.

6. In a video processing system having a video encoding device and a video decoding device, The aforementioned video encoding device is A downscaling unit generates a first video signal of a first resolution video and a second video signal of a second resolution video with a higher resolution than the first resolution video by downscaling the third video signal of a third resolution video. The system includes a video encoding unit that performs spatially scalable encoding on the first video signal and the second video signal, and outputs the encoded first video signal and the second video signal. The aforementioned video decoding device is A video decoding unit that decodes the encoded first video signal and the second video signal by spatially scalable decoding, The system includes an upscaling unit that generates a third video signal relating to a third resolution video with a higher resolution than the second resolution video by upscaling the first video signal and the second video signal using a neural network, and outputs the third video signal. The upscaling unit is, A first upscaling module that upscales the first video signal using the first neural network of the aforementioned neural network, and a first feature extraction module that extracts feature quantities from the upscaled first video signal using the second neural network of the aforementioned neural network, A second upscaling module generates a third video signal by upscaling the second video signal using the features of the first video signal with the first neural network, has Video processing system.

Images