Encoding device, encoding program, pre-processing device, and pre-processing program

Abstract

To provide an encoding device and an encoding program that are capable of improving not only subjective image quality, but also objective image quality.SOLUTION: An encoding device 100 includes a pre-processing unit 110 that performs pre-processing on a first video signal using a trained first neural network model, and outputs the pre-processed video signal. The encoding device 100 also includes a video encoding unit 120 that performs encoding processing on the pre-processed video signal using a picture reference structure, and outputs the encoded pre-processed video signal. Here, the first neural network model is trained using a neural network video encoding unit including a second neural network model trained using the same picture reference structure as the picture reference structure used in the video encoding unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an encoding device, an encoding program, a pre-processing device, and a pre-processing program. [Background technology]

[0002] In recent years, coding methods using neural networks have been attracting attention.

[0003] For example, there is an encoding method that uses a CNN (Convolutional Neural Network) to perform a motion detection process that detects a motion vector between a current image and a past image, a motion compensation process that generates a predicted image from the motion vector, and a conversion process that converts (e.g., DCT conversion) a difference image between an original image and a predicted image (e.g., Non-Patent Document 1). By optimizing such an encoding method end-to-end, it is possible to obtain encoding efficiency equivalent to that of a conventional encoding method such as HEVC (High Efficiency Video Coding).

[0004] In addition to the coding method using a neural network, there is also a coding method that performs preprocessing before coding (for example, Non-Patent Documents 2 and 3). Regarding the coding method using preprocessing, it has been reported that the coding efficiency is improved when a subjective image quality index based on MS-SSIM (Multi-Scale Structural Similarity) or VMAF (Video Multimethod Assessment Fusion) is used, compared to conventional coding methods such as H.264. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai and Z. Gao, "DVC: An End-To-End Deep Video Compression Framework," 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 April 7, pp. 10998-11007. [Non-Patent Document 2] A. Chadha and Y. Andreopoulos, "Deep Perceptual Preprocessing for Video Coding," 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14847-14856. [Non-Patent Document 3] A. Chadha, R. Anam, I. Fadeev, V. Giotsas, and Y. Andreopoulos, “Escaping The Complexity-Bitrate-Quality Barriers Of Video Encoders Via Deep Perceptual Optimization,” SPIE2020, Optical Engineering + Applications. Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present disclosure is to provide an encoding device, an encoding program, a pre-processing device, and a pre-processing program that are capable of improving not only the subjective image quality but also the objective image quality. [Means for solving the problem]

[0007] The encoding device according to the first aspect includes a pre-processing unit that performs pre-processing on a first video signal using a trained first neural network model and outputs the pre-processed video signal. The encoding device also includes a video encoding unit that performs encoding on the pre-processed video signal using a picture reference structure and outputs the encoded encoded video signal. Here, the first neural network model is trained using a neural network video encoding unit that includes a second neural network model trained using the same picture reference structure as the picture reference structure used in the video encoding unit.

[0008] A coding program according to a second aspect is a coding program executed by a computer of a coding device. The coding program causes the computer to execute a process of performing pre-processing on a first video signal using a trained first neural network model, and outputting the pre-processed video signal. The coding program also causes the computer to execute a process of performing coding on the pre-processed video signal using a picture reference structure, and outputting the coded coded video signal. Here, the learning of the first neural network model is performed using a neural network video coding unit including a second neural network model trained using the same picture reference structure as the picture reference structure used in the process of outputting the coded video signal.

[0009] A pre-processing device according to a third aspect includes a pre-processing unit that performs pre-processing on a first video signal using a trained first neural network model and outputs the pre-processed video signal to a coding device. The coding device performs coding processing on the pre-processed video signal using a picture reference structure. The first neural network model is trained using a neural network video coding unit including a second neural network model trained using the same picture reference structure as the picture reference structure used by the coding device.

[0010] A pre-processing program according to a fourth aspect is a pre-processing program executed by a computer of a pre-processing device. The pre-processing program causes the computer to execute a process of performing pre-processing on a first video signal using a trained first neural network model, and outputting the pre-processed video signal to a coding device. The coding device performs coding processing on the pre-processed video signal using a picture reference structure. The first neural network model is trained using a neural network video coding unit including a second neural network model trained using the same picture reference structure as the picture reference structure used in the coding device. Effect of the Invention

[0011] According to the present disclosure, it is possible to provide an encoding device, an encoding program, a pre-processing device, and a pre-processing program that are capable of improving not only the subjective image quality but also the objective image quality. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an encoding device at the time of inference according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of an encoding device during learning according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of a video encoding unit according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a picture reference structure according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of the configuration of a neural network video encoding unit according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of an encoding device during learning according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of an operation during learning according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of an operation during inference according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of an experimental result according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of an encoding system at the time of inference according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of the configuration of an encoding system during learning according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] [First embodiment]

[0014] (Encoding method using neural networks) As described above, encoding methods using neural networks have been proposed. Here, the encoding method according to Non-Patent Document 1 and the encoding method according to Non-Patent Document 2 will be described.

[0015] (Encoding method according to Non-Patent Document 1) In Non-Patent Document 1, the coding method using a neural network is called DVC (Deep Video Compression). In DVC, coding processing is performed according to the following procedure.

[0016] Step N1 (motion detection): The current frame and the decoded past frame are input, and an optical flow value (i.e., motion information) is obtained using a CNN (optical flow network). Then, the motion information is compressed and decoded using a CNN (MV encoder network and MV decoder network) or the like to obtain quantized motion information.

[0017] Step N2 (motion compensation): Using the quantized motion information and the past frame as input, a predicted frame is obtained using a CNN (motion compensation network).

[0018] Step N3-N4 (transformation, quantization, and inverse transformation): A residual frame between the current frame and the predicted frame is output using a CNN (residual encoder network), and the output is quantized to obtain a quantized value. A reconstructed residual frame is obtained from the quantized value using a CNN (residual decoder network).

[0019] Step N5 (entropy coding): The quantized motion information obtained in step N1 and the quantized values ​​obtained in steps N3-N4 are used as inputs to obtain coded bits using a CNN (bitrate estimation network).

[0020] Step N6 (frame reconstruction): The predicted frame obtained in step N2 is added to the reconstructed residual frame obtained in steps N3-N4 to obtain a reconstructed frame (decoded frame).

[0021] The above is the encoding procedure by DVC. In DVC, during learning, a loss function (λD+R) is used using the distortion D between the current frame and the decoded frame obtained in step N6 and the number of bits used for encoding (input in step N6) R. In DVC, it is possible to construct an end-to-end optimal neural network encoding model by using the loss function (λD+R) to find parameters (or solve an optimization problem) that minimize the number of bits R and minimize the distortion D.

[0022] (Encoding method according to Non-Patent Document 2) Also, as described above, an encoding method (for example, Non-Patent Document 2) has been proposed that performs pre-processing before encoding. In Non-Patent Document 2, pre-processing is called DPP (Deep Perceptual Preprocessing). In an encoding method using DPP, during learning, the main components of encoding are virtualized, DPP is trained, and DPP is optimized using a loss function. Then, during inference, pre-processing is performed on an input image using the optimized DPP, and encoding processing is performed using an existing codec on the image after pre-processing.

[0023] In the encoding method using DPP, even if the encoding process is performed with any codec at any bit rate, it is possible to obtain an optimized DPP that balances perception and distortion using a loss function. Therefore, even if an image encoded with the encoding method using DPP is decoded, the visual quality can be improved compared to when DPP is not performed. In other words, the encoding method using DPP can improve the encoding efficiency (better image quality even with the same amount of data) compared to encoding methods that do not use DPP.

[0024] The above is an example of an encoding method using a neural network.

[0025] There are two methods for evaluating image quality: subjective image quality and objective image quality.

[0026] The subjective image quality is, for example, image quality evaluated based on the senses of a person who sees an image. Therefore, while the subjective image quality can be evaluated in accordance with human senses, it may vary depending on the evaluation method. The coding method using the neural network described above aims to improve the subjective image quality by using a subjective image quality index such as SSIM or VMAF.

[0027] On the other hand, the objective image quality is, for example, image quality that quantitatively evaluates image quality degradation caused by encoding by signal analysis. Objective image quality evaluation indexes include SNR (Signal-to-Noise Ratio), which indicates the ratio of signal to noise, and PSNR (Peak Signal-to-Noise Ratio), which indicates the ratio of noise to maximum power. Video encoding standards such as HEVC (High Efficiency Video Codec) and VVC (Versatile Video Coding) mainly aim to maximize PSNR (Peak signal-to-noise ratio).

[0028] Generally, subjective image quality and objective image quality may not agree. For example, assume that image A is compared with image B, which is shifted one pixel to the right from image A. In this case, the subjective image quality is only shifted by one pixel overall, and the evaluation of image B relative to image A is not significantly different. On the other hand, in the case of objective image quality, the difference in pixel values ​​between the two images at the same pixel position becomes large, so for example the PSNR will be a very small value.

[0029] In this way, although coding techniques using neural networks can improve subjective image quality, they may not necessarily be able to improve objective image quality because the subjective image quality and the objective image quality may not coincide.

[0030] Therefore, the first embodiment aims to improve not only the subjective image quality but also the objective image quality.

[0031] Hereinafter, the first embodiment will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0032] (1. Example of the configuration of the encoding device during inference) Next, an example of the configuration of an encoding device will be described.

[0033] FIG. 1 is a diagram illustrating an example of the configuration of an encoding device according to the first embodiment.

[0034] As shown in Fig. 1, the encoding device 100 includes a pre-processing unit 110 and a video encoding unit 120. Here, the encoding device 100 shown in Fig. 1 represents an example of the configuration of the encoding device 100 at the time of inference.

[0035] That is, the pre-processing unit 110 includes a neural network model (for example, a first neural network model) that has already been trained, and the pre-processing is performed using the trained model. The pre-processing may be, for example, a process of upscaling the resolution of the input video, or a process of downscaling the resolution of the input video. Alternatively, the pre-processing may be a process that does not change the resolution. The pre-processing may be any process that improves both the subjective image quality and the objective image quality, as compared with the conventional pre-processing described in Non-Patent Document 2 and the like. However, the pre-processing unit 110 performs the pre-processing using the same picture reference structure as the picture reference structure used in the video encoding unit 120. In the first embodiment, for example, as shown in FIG. 4, the MPEG2 picture reference structure (GOP (Group Of Picture) structure with 15 intra periods and 2 consecutive frames) is used. That is, the pre-processing unit 110 inputs a video signal (e.g., a first video signal) for each GOP size of 16 video frames, performs pre-processing on the video signal using a trained neural network model (e.g., a first neural network model), and outputs the pre-processed video signal.

[0036] The video encoding unit 120 performs encoding processing on the pre-processed video signal, for example, by using the picture reference structure shown in Fig. 4, and outputs the encoded encoded video signal. The encoding method in the video encoding unit 120 may be any method of the video encoding standard, such as HEVC or VVC, but the first embodiment will be described assuming that MPEG2 is used. During inference, the video encoding unit 120 performs encoding processing using the video encoding standard used during learning. Furthermore, during inference, the video encoding unit 120 performs encoding processing using the same picture reference structure as the picture reference structure used during learning.

[0037] In the first embodiment, first, the neural network video encoding unit is trained, and then the trained neural network video encoding unit is used to train the pre-processing unit 110. Here, it is also possible to train the pre-processing unit 110 using the video encoding unit 120. However, in the first embodiment, the pre-processing unit 110 is not trained using the video encoding unit 120, but is trained using the trained neural network video encoding unit. The reason for this is, for example, as follows.

[0038] That is, in the video encoding unit 120, a quantization process is performed on a signal after conversion (here, DCT conversion process), but a part of the signal components may be deleted by rounding the quantized signal. Since such rounding is not differentiable, the error backpropagation method, which is a learning method for neural networks, cannot be used. On the other hand, in the neural network video encoding unit, such rounding is not performed, but is replaced by adding random noise, thereby reproducing the quantization process and the like using a neural network. Therefore, it is expected that the learning result for the pre-processing unit 110 will be better when the neural network video encoding unit is used than when the video encoding unit 120 is used.

[0039] In the following, as a configuration example of the encoding device 100 during training, (2.1) a configuration example of the encoding device 100 when training is performed on the neural network video encoding unit will be described. Next, (2.2) a configuration example of the encoding device 100 when training is performed on the pre-processing unit 110 using a trained neural network video encoding unit will be described.

[0040] In the following, there may be cases where the terms "neural network model" and "neural network" are used without distinction. For example, there may be cases where "performing pre-processing using a neural network model" and "performing pre-processing using a neural network" are used without distinction.

[0041] In addition, the pre-processing unit 110 has a neural network model (e.g., a first neural network model), and learning or inference is performed on the neural network model, but there are cases where the terms "learning and inference on the neural network model included in the pre-processing unit" and "learning and inference on the pre-processing unit 110" are used interchangeably.

[0042] Furthermore, the neural network video encoding unit 130 also has a neural network model (e.g., a second neural network model), and learning or inference is performed on the neural network model. However, there are cases where the terms "learning and inference on the neural network model included in the neural network video encoding unit 130" and "learning and inference on the neural network video encoding unit 130" are used interchangeably.

[0043] Furthermore, there are cases where "processing on video" and "processing on video signals" are used without distinction. For example, there are cases where "encoding processing on input video" and "encoding processing on video signals of input video" are used without distinction.

[0044] (2. Example of the configuration of the encoding device during learning) (2.1 Example of the configuration of the encoding device when learning is performed in the neural network video encoding unit) Fig. 2 is a diagram illustrating an example of the configuration of the encoding device according to the first embodiment. Fig. 2 illustrates an example of the configuration of the encoding device 100 when training is performed on the neural network video encoding unit 130.

[0045] As shown in FIG. 2, the encoding device 100 includes a video encoding unit 120, a neural network video encoding unit 130, and an error determining unit 140 (eg, a first error determining unit).

[0046] The video encoding unit 120 may be the same as the video encoding unit 120 (FIG. 1) used during inference. The video encoding unit 120 performs encoding processing on a video signal of an input video (for example, a second video signal) using a picture reference structure, and outputs an encoded encoded video signal. The encoding itself may use a method according to a video encoding standard, and in the first embodiment, it is assumed that MPEG2 is used as described above.

[0047] Fig. 3 is a diagram illustrating an example of the configuration of the video encoding unit 120 according to the first embodiment. As illustrated in Fig. 3, the video encoding unit 120 has an intra encoding unit 121 and an inter encoding unit 122. The video encoding unit 120 performs encoding processing on the video signal of each video frame in accordance with the picture reference structure illustrated in Fig. 4.

[0048] That is, the first video frame in the GOP is treated as an I frame, and the intra coding unit 121 performs intra coding processing. The intra coding unit 121 outputs a video signal of the coded I frame. The second to sixteenth video frames in the GOP are input to the inter coding unit 122 as B frames or P frames according to the picture reference structure shown in FIG. 4, and inter coding processing is performed sequentially. The inter coding unit 122 outputs a video signal of the coded B frame or a video signal of the coded P frame. The video signals of the coded I frame, the coded B frame, and the coded P frame are output as video signals of coded video according to the temporal order shown in the picture reference structure shown in FIG. 4.

[0049] Returning to FIG. 2, the video encoding unit 120 outputs a video signal of the encoded video to the error determining unit 140.

[0050] The neural network video encoding unit 130 performs intra-coding and inter-coding on an input video signal (e.g., a second video signal) using a neural network model (e.g., a second neural network model), and outputs a neural network-encoded video signal (hereinafter, sometimes referred to as a “neural network-encoded video signal”). The neural network video encoding unit 130 outputs the neural network-encoded video signal to the error determining unit 140.

[0051] Fig. 5 is a diagram illustrating an example of the configuration of the neural network video encoding unit 130. As illustrated in Fig. 5, the neural network video encoding unit 130 includes a neural network intra encoding unit 131, a neural network inter encoding unit 132, and a neural network code amount calculation unit 133.

[0052] The neural network video encoder 130 also uses the same picture referencing structure as that used by the video encoder 120 to perform the encoding process using a neural network.

[0053] That is, the first video frame in a GOP is input as an I-frame to the neural network intra-encoding unit 131. The neural network intra-encoding unit 131 performs intra-encoding processing on the I-frame using a neural network model. The neural network intra-encoding unit 131 outputs a video signal of the encoded I-frame.

[0054] While any neural network model may be used for the I-frame, the first embodiment will be described assuming that intra-encoding processing is performed by the encoding method described in Non-Patent Document 4 below. In Non-Patent Document 4, for example, input video is subjected to parameter conversion using a neural network including a convolution layer or the like to convert it into latent variables, and the latent variables are further subjected to parameter conversion using a neural network to convert them into latent variables in which standard deviations are summarized. Then, quantization and arithmetic conversion are performed on the latent variables.

[0055] Non-Patent Document 4: Johannes Balle, David Minnen, Saurabh Singh, Sung Jin Hwang, Nick Johnston: “Variational image compression with a scale hyperprior”, 2018; arXiv:1802.01436. Next, the second to sixteenth video frames in the GOP are input to the neural network inter-encoding unit 132 as B frames or P frames according to the picture reference structure shown in Fig. 4. In the neural network inter-encoding unit 132, an inter-encoding process using a neural network is sequentially performed on the video signal of each video frame. The neural network inter-encoding unit 132 outputs the video signal of the coded B frame and the video signal of the coded P frame. The video signal of each coded video frame is output as a video signal of neural network-coded video according to the temporal order shown in the picture reference structure shown in Fig. 4.

[0056] Note that, although any model may be used as the neural network model for the B and P frames, the first embodiment will be described assuming that DVC corresponding to bidirectional prediction and described in Non-Patent Document 1 is used. Alternatively, the neural network model for the B and P frames may be a neural network model performing motion compensation for multiple frames, as described in Non-Patent Document 5 below.

[0057] Non-patent document 5: M. Akin Yilmaz, A. Murat Tekalp: “End-to-End Rate-Distortion Optimized Learned Hierarchical Bi-Directional Video Compression”, 2021; arXiv:2112.09529. The neural network code amount calculation unit 133 receives the video signal of the neural network encoded video, calculates the code amount of the video signal, and outputs the calculated code amount as the neural network code amount.

[0058] 5, the neural network video encoding unit 130 has a neural network model used in the neural network intra-encoding unit 131 and a neural network model used in the neural network inter-encoding unit 132. Although the two neural network models are different, the neural network models included in the neural network video encoding unit 130 may be described as one neural network model below. That is, the neural network model used in the neural network intra-encoding unit 131 and the neural network model used in the neural network inter-encoding unit 132 may be combined and described as one neural network model in the neural network video encoding unit 130 (e.g., a second neural network model).

[0059] Returning to Fig. 2, error determination unit 140 performs error determination between the neural network encoded video signal, which is the output of neural network video encoding unit 130 for a video signal of an input video (e.g., a second video signal), and the encoded video signal, which is the output of video encoding unit 120 for the video signal of the input video, and optimizes parameters of neural network video encoding unit 130. Specifically, error determination unit 140 receives the neural network encoded video signal, the encoded video signal, and the neural network code amount. Then, error determination unit 140 sets parameters of the neural network (e.g., a second neural network model) used in neural network video encoding unit 130 so that the following loss function L1 is minimized.

[0060]

number

[0061] In this way, the neural network video coding unit 130 can derive a trained neural network model with optimized parameters when training its own neural network model. At this time, the (neural network model of) the neural network video coding unit 130 generates neural network-coded video using the same picture reference structure as the picture reference structure used in the video coding unit 120, and optimizes the parameters. The video coding unit 120 performs processing with the aim of improving objective image quality, and the neural network video coding unit 130 optimizes the parameters using the same picture reference structure as the picture reference structure used in the video coding unit 120. Therefore, the neural network model in the neural network video coding unit 130 is a trained model that improves not only subjective image quality but also objective image quality, as compared with, for example, DVC in Non-Patent Document 1.

[0062] Next, in the encoding device 100, the neural network model (eg, the first neural network model) of the pre-processing unit 110 is trained using the neural network video encoding unit 130 having the trained model.

[0063] (2.2 Example of the configuration of the encoding device when learning is performed in the pre-processing unit 110) FIG. 6 is a diagram illustrating an example of the configuration of the encoding device 100 when learning is performed in the pre-processing unit 110. As shown in FIG.

[0064] As shown in FIG. 6, the encoding device 100 includes a pre-processing unit 110, a neural network video encoding unit 130, and an error determining unit 145 (eg, a second error determining unit).

[0065] The pre-processing unit 110 performs pre-processing on the video signal of the input video (for example, the third video signal) and outputs the pre-processed video signal after pre-processing. The pre-processing unit 110 performs pre-processing by using the same picture reference structure as the picture reference structure used in the video encoding unit 120. For example, the pre-processing unit 110 uses the same picture reference structure as the GOP structure of MPEG2 shown in FIG. 4. The pre-processing unit 110 inputs the video signal of each of the 16 video frames, with 16 video frames as one GOP, and outputs the video signal of each of the 16 video frames. The pre-processing unit 110 performs pre-processing by using a neural network model (for example, a first neural network model). Any model may be used as the neural network model in the pre-processing unit 110, but in the first embodiment, BasicVSR shown in the following non-patent document 6 is used. BasicVSR is a neural network model that can take into account time-series correlation. BasicVSR is a Video Super Resolution (VSR) neural network model that expands the number of pixels, but by changing the final processing stage within the model, it is possible to output images with any resolution.

[0066] Non-patent document 6: Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, Chen Change Loy: “BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond”, 2020; arXiv:2012.02181. The neural network video encoding unit 130 outputs a neural network-encoded video signal using a trained neural network model for the pre-processed video signal. The neural network video encoding unit 130 uses the same picture reference structure as the picture reference structure used by the video encoding unit 120. For example, the neural network video encoding unit 130 performs encoding processing on each of the 16 pre-processed video frames using a neural network, and outputs each of the 16 neural network-encoded video frames. The neural network video encoding unit 130 outputs the neural network-encoded video signal to the error determination unit 145. In addition, the neural network video encoding unit 130 calculates the code amount of the neural network-encoded video signal, and outputs it to the error determination unit 145 as the neural network code amount. The calculation of the neural network code amount is performed using the neural network code amount calculation unit 133 (FIG. 5). Note that an example of the configuration of the neural network video encoding unit 130 when learning is performed in the pre-processing unit 110 may be the example of the configuration shown in FIG. 5.

[0067] The error determination unit 145 performs error determination between the neural network encoded video signal, which is the output of the neural network video encoding unit 130 for the pre-processed video signal, and the video signal of the input video input to the pre-processing unit 110, and optimizes the parameters of the pre-processing unit 110. Specifically, the error determination unit 145 inputs the neural network encoded video signal, the video signal of the input video, and the neural network code amount. Then, the error determination unit 145 sets parameters of the neural network (e.g., a first neural network model) used in the pre-processing unit 110 so that the following loss function L2 is minimized.

[0068]

number

[0069] As described above, in the first embodiment, the neural network model (e.g., the first neural network model) in the pre-processing unit 110 is trained using the neural network video encoding unit 130 including a neural network model (e.g., the second neural network model) trained using the same picture reference structure as the video encoding unit 120. Therefore, in the first embodiment, it is possible to derive a trained model in the neural network model in the pre-processing unit 110 that improves not only the subjective image quality but also the objective image quality, as compared with the model in Non-Patent Document 2, for example.

[0070] In addition, since the pre-processing unit 110 performs pre-processing different from the encoding process during learning, it is possible to learn inter-frame features that are not referenced in the picture reference structure used in encoding. This makes it possible to perform pre-processing that reduces the encoding difficulty to a certain level or lower.

[0071] 6, when the pre-processing in the pre-processing unit 110 includes upscaling, the neural network video and the input video can be made to have the same resolution by performing downscaling on the neural network encoded video signal. When the pre-processing includes downscaling, the neural network video and the input video can be made to have the same resolution by performing upscaling on the neural network encoded video signal. The process of performing downscaling or upscaling on the neural network encoded video signal may be performed by the neural network video encoding unit 130 or the error determination unit 145, for example.

[0072] (Operation example according to the first embodiment) Next, an operation example according to the first embodiment will be described.

[0073] FIG. 7 is a diagram illustrating an example of an operation during learning according to the first embodiment.

[0074] As shown in FIG. 7, in step S10, the encoding device 100 starts the process.

[0075] In step S11, the neural network video encoding unit 130 receives a video signal of an input video, and outputs a video signal of a neural network-encoded video. Also in step S11, the video encoding unit 120 receives a video signal of an input video, and outputs a video signal of an encoded video. In the neural network video encoding unit 130, the same picture reference structure as that used in the encoding process of the video encoding unit 120 is used to cause learning of the neural network.

[0076] In step S12, the error determining unit 140 performs error determination between the neural-network-encoded video and the encoded video using Equation 1, and optimizes parameters for the neural network in the neural-network video encoding unit 130. In the neural-network video encoding unit 130, a trained neural network model is derived.

[0077] In step S13, the pre-processing unit 110 performs pre-processing on the input video using a neural network model (e.g., a first neural network model), and outputs the pre-processed video signal to the neural network video coding unit 130. The neural network video coding unit 130 performs coding on the pre-processed video signal using a trained neural network model (e.g., a second neural network model), and outputs a neural network-coded video signal.

[0078] In step S14, the error determining unit 145 determines an error between the neural-network-encoded image and the input image, and optimizes parameters of the neural network model of the pre-processing unit 110. In the pre-processing unit 110, a trained neural network model is derived.

[0079] In step S15, the encoding device 100 ends the learning.

[0080] FIG. 8 is a diagram illustrating an example of an operation during inference according to the first embodiment.

[0081] As shown in FIG. 8, in step S20, the encoding device 100 starts inference.

[0082] In step S21, the pre-processing unit 110 performs pre-processing on the input video using a trained neural network model, and outputs a pre-processed video signal.

[0083] In step S22, the video encoding unit 120 performs encoding processing on the pre-processed video signal in accordance with a video encoding standard, and outputs an encoded video signal. The video encoding unit 120 may transmit the encoded video signal to the video decoding device as a bit stream.

[0084] In step S23, the encoding device 100 ends the inference.

[0085] (Experimental Results) Next, experimental results relating to the first embodiment will be described.

[0086] Fig. 9 is a diagram showing an example of an experimental result, which shows the results of the learning and inference described in the first embodiment.

[0087] The input images used were "controlled_burn", "crowd_run", "ducks_take_off", and "park_joy", all of which are distributed by the Xiph.Org Foundation (XIPH) on the Internet (registered trademark). All images were HD (High Definition) images. The images were encoded using the free software "ffmpeg" in MPEG2, with the "qscale" parameter, which indicates image quality, set to "12" (where "1" indicates the highest quality and "32" indicates the lowest), "17", "22", and "27". PSNR indicates objective image quality, and VMAF indicates subjective image quality. For all image quality values, the change in BD-Rate due to the presence or absence of pre-processing was measured. For both PSNR and VMAF, the more negative the value, the better the encoding efficiency.

[0088] 9, an experimental result was obtained in which the learning and inference according to the first embodiment improves not only the subjective image quality (VMAF) but also the objective image quality (PSNR) compared to when pre-processing is not performed. The experimental result also shows that the learning and inference described in the first embodiment makes it possible to improve not only the subjective image quality but also the objective image quality compared to when pre-processing is not performed.

[0089] [Second embodiment] Next, a second embodiment will be described.

[0090] In the first embodiment, an example has been described in which the encoding device 100 includes the pre-processing unit 110. In the second embodiment, an example will be described in which a pre-processing device having the pre-processing unit 110 is provided in the front stage of the encoding device 100.

[0091] FIG. 10 is a diagram illustrating an example of the configuration of an encoding system 10 according to the second embodiment.

[0092] 10, the encoding system 10 includes a pre-processing device 200 and an encoding device 100. The pre-processing device 200 includes a pre-processing unit 110. In addition, the encoding device 100 includes a video encoding unit 120.

[0093] The encoding system 10 shown in Fig. 10 shows an example of a configuration at the time of inference, similar to Fig. 1. That is, the pre-processing unit 110 performs pre-processing using an already trained neural network model (e.g., a first neural network model). Specifically, the pre-processing unit 110 performs pre-processing on a video signal of an input video (e.g., a first video signal) using the trained neural network model, and outputs the pre-processed video signal to the encoding device 100.

[0094] As in the first embodiment, the encoding device 100 performs encoding processing on the pre-processed video signal by using the picture referencing structure of the video encoding standard, and outputs a video signal of encoded video.

[0095] FIG. 11 is a diagram illustrating an example of the configuration of the encoding system 10 during learning according to the second embodiment.

[0096] As shown in FIG. 11, during learning, the pre-processing device 200 has a pre-processing unit 110, and the encoding device 100 has a neural network video encoding unit 130 and an error determining unit 145.

[0097] As in the first embodiment (FIG. 6), the pre-processing unit 110 performs pre-processing on the video signal of the input video by using a neural network model (eg, a first neural network model), and outputs the pre-processed video signal.

[0098] As in the first embodiment, the neural network video encoding unit 130 has a trained neural network model (e.g., a second neural network model) that has been trained using the same picture reference structure as the picture reference structure used in the encoding process of the video encoding unit 120. As in the first embodiment, the neural network video encoding unit 130 performs encoding processing on the video signal of the pre-processed video using the neural network model, and outputs the video signal of the neural network-encoded video.

[0099] Similarly to the first embodiment, the error determining unit 145 also optimizes the parameters of the neural network used in the pre-processing unit 110 using the loss function L2 shown in Equation 2.

[0100] That is, in the second embodiment, as in the first embodiment, training of the neural network model in the pre-processing unit 110 is performed using the neural network video encoding unit 130 including a neural network model trained using the same picture reference structure as the picture reference structure used in the encoding device 100.

[0101] Therefore, in the second embodiment, as in the first embodiment, the pre-processing unit 110 is trained using the neural network video encoding unit 130 trained using the same picture reference structure as the video encoding standard. Therefore, in the second embodiment, as in the first embodiment, it is possible to improve not only the subjective image quality but also the objective image quality as compared to the above-mentioned DPP and the like.

[0102] [Other embodiments] In the first and second embodiments, an example has been described in which the pre-processing unit 110 is trained after the neural network video encoding unit 130 is trained, but after the pre-processing unit 110 is trained, the neural network video encoding unit 130 may be trained again, and the pre-processing unit 110 may then be trained again. In other words, the neural network video encoding unit 130 may be trained and then the pre-processing unit 110 may be trained repeatedly.

[0103] A program may be provided that causes a computer to execute each process performed by the above-mentioned devices (encoding device 100 and pre-processing device 200). The program may be recorded in a computer-readable medium. The computer-readable medium can be used to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Furthermore, circuits that execute each process performed by the above-mentioned devices (encoding device 100 and pre-processing device 200) may be integrated, and the device may be configured as a semiconductor integrated circuit (chip set, SoC).

[0104] Although the embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes can be made without departing from the scope of the invention. In addition, each operation example can be combined without any contradiction. [Explanation of symbols]

[0105] 100: Encoding device 110: Pre-processing unit 120: Video encoding unit 121: Intra-coding unit 122: Inter-encoding unit 130: Neural network video encoding unit 131: Neural network intra coding unit 132: Neural network inter-encoding unit 133: Neural network code amount calculation unit 140, 145: Error determination unit 200: Pre-processing device

Claims

1. a pre-processing unit that performs pre-processing on the first video signal by using the trained first neural network model and outputs the pre-processed video signal; a video encoding unit that performs an encoding process on the pre-processed video signal by using a picture reference structure, and outputs an encoded video signal; The first neural network model is trained using a neural network video encoding unit including a second neural network model trained using the same picture reference structure as the picture reference structure used in the video encoding unit. Encoding device.

2. The neural network video encoding unit performs intra-encoding and inter-encoding on the second video signal using the second neural network model, and outputs a neural network-encoded video signal.

2. The encoding device according to claim 1.

3. a first error determination unit that performs error determination between a neural network encoded video signal that is an output of the neural network video encoding unit for the second video signal and an encoded video signal that is an output of the video encoding unit for the second video signal when the second neural network model is trained, and optimizes parameters of the second neural network model.

2. The encoding device according to claim 1.

4. and a second error determination unit that performs error determination between a neural network-encoded video signal, which is an output of the neural network video encoding unit for the pre-processed video signal, and a third video signal input to the pre-processing unit, when the first neural network model is trained, and optimizes parameters of the first neural network model.

2. The encoding device according to claim 1.

5. The pre-processing unit performs the pre-processing on the first video signal by using the same picture reference structure as that of the video encoding unit.

2. The encoding device according to claim 1.

6. An encoding program to be executed by a computer of an encoding device, performing pre-processing on the first video signal using the trained first neural network model, and outputting the pre-processed video signal; a process of encoding the pre-processed video signal by using a picture reference structure, and outputting the encoded video signal; The first neural network model is trained using a neural network video encoding unit including a second neural network model trained using the same picture reference structure as the picture reference structure used in the process of outputting the encoded video signal. Encoding program.

7. A pre-processing device, a pre-processing unit that performs pre-processing on the first video signal by using the trained first neural network model and outputs the pre-processed video signal to the encoding device; In the encoding device, an encoding process is performed on the pre-processed video signal by using a picture reference structure, The first neural network model is trained using a neural network video encoding unit including a second neural network model trained using the same picture reference structure as the picture reference structure used in the encoding device. Pre-processing device.

8. A pre-processing program executed by a computer of a pre-processing device, performing pre-processing on the first video signal using the trained first neural network model, and outputting the pre-processed video signal to an encoding device; In the encoding device, an encoding process is performed on the pre-processed video signal by using a picture reference structure, The first neural network model is trained using a neural network video encoding unit including a second neural network model trained using the same picture reference structure as the picture reference structure used in the encoding device. Pre-processing program.

Images