Encoding device, encoding program, preprocessing device, and preprocessing program

Abstract

To provide an encoding device, an encoding program, a pre-processing device, and a pre-processing program that have a video encoding model and a pre-processing model with a certain level of accuracy or higher level.SOLUTION: An encoding device 100 includes a pre-processing unit 110 that performs pre-processing on a first video signal using a trained first neural network model, and outputs the pre-processed video signal. The encoding device 100 also includes a video encoding unit 120 that performs encoding processing on the pre-processed video signal, and outputs the encoded pre-processed video signal. The first neural network model is trained using a neural network video encoding unit 130 including a trained second neural network model. The second neural network model is trained using a video signal of a first predicted image generated when the video encoding unit 110 encodes the second video signal and a video signal of a second predicted image generated when the neural network video encoding unit 130 encodes the second video signal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an encoding device, an encoding program, a pre-processing device, and a pre-processing program. [Background technology]

[0002] In recent years, coding methods using neural networks have been attracting attention.

[0003] For example, there is an encoding method that uses a CNN (Convolutional Neural Network) to perform a motion detection process that detects a motion vector between a current image and a past image, a motion compensation process that generates a predicted image from the motion vector, and a conversion process that converts (e.g., DCT conversion) a difference image between an original image and a predicted image (e.g., Non-Patent Document 1). By optimizing such an encoding method end-to-end, it is possible to obtain encoding efficiency equivalent to that of a conventional encoding method such as HEVC (High Efficiency Video Coding).

[0004] In addition to the coding method using a neural network, there is also a coding method that performs preprocessing before coding (for example, Non-Patent Documents 2 and 3). Regarding the coding method using preprocessing, it has been reported that the coding efficiency is improved when a subjective image quality index based on MS-SSIM (Multi-Scale Structural Similarity) or VMAF (Video Multimethod Assessment Fusion) is used, compared to conventional coding methods such as H.264.

[0005] Furthermore, there is a coding method that performs pre-processing using a neural network, a neural network that imitates a video coding standard (called an "auxiliary codec network"), a neural network that predicts the amount of bits generated end-to-end, and post-processing using a neural network on the restored image (for example, Non-Patent Document 4). Non-Patent Document 4 reports on learning to make the auxiliary codec network imitate the video coding standard while suppressing the overall bit rate. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai and Z. Gao, "DVC: An End-To-End Deep Video Compression Framework," 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 April 7, pp. 10998-11007. [Non-Patent Document 2] A. Chadha and Y. Andreopoulos, "Deep Perceptual Preprocessing for Video Coding," 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14847-14856. [Non-Patent Document 3] A. Chadha, R. Anam, I. Fadeev, V. Giotsas, and Y. Andreopoulos, “Escaping The Complexity-Bitrate-Quality Barriers Of Video Encoders Via Deep Perceptual Optimization,” SPIE2020, Optical Engineering + Applications. [Non-Patent Document 4] Hanbin Son, Taeoh Kim, Hyeongmin Lee, Sangyoun Lee: “Enhanced Standard Compatible Image Compression Framework based on Auxiliary Codec Networks”, 2020; arXiv:2009.14754. Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present disclosure is to provide an encoding device, an encoding program, a pre-processing device, and a pre-processing program that have a video encoding model and a pre-processing model with a certain level of accuracy or higher. [Means for solving the problem]

[0008] The encoding device according to the first aspect has a pre-processing unit that performs pre-processing on a first video signal using a trained first neural network model and outputs a pre-processed video signal. The encoding device also has a video encoding unit that performs encoding on the pre-processed video signal and outputs an encoded encoded video signal. The first neural network model is trained using a neural network video encoding unit including a trained second neural network model. The second neural network model is trained using a video signal of a first predicted image generated when the video encoding unit encodes the second video signal and a video signal of a second predicted image generated when the neural network video encoding unit encodes the second video signal.

[0009] A coding program according to a second aspect is a coding program executed by a computer of a coding device. The coding program causes the computer to execute a process of performing pre-processing on a first video signal using a trained first neural network model, and outputting a video signal after the pre-processing. The coding program also causes the computer to execute a process of encoding the video signal after the pre-processing, and outputting an encoded video signal. The first neural network model is trained using a neural network video encoding unit including a trained second neural network model. The second neural network model is trained using a video signal of a first predicted image generated when the video encoding unit encodes the second video signal, and a video signal of a second predicted image generated when the neural network video encoding unit encodes the second video signal.

[0010] A pre-processing device according to a third aspect has a pre-processing unit that performs pre-processing on a first video signal using a trained first neural network model and outputs the pre-processed video signal to an encoding device. The encoding device performs encoding on the pre-processed video signal and outputs an encoded encoded video signal. The first neural network model is trained using a neural network video encoding unit including a trained second neural network model. The second neural network model is trained using a video signal of a first predicted image generated when the video encoding unit encodes the second video signal and a video signal of a second predicted image generated when the neural network video encoding unit encodes the second video signal.

[0011] A pre-processing program according to a fourth aspect is a pre-processing program executed by a computer of a pre-processing device. The pre-processing program causes the computer to execute a process of performing pre-processing on a first video signal using a trained first neural network model, and outputting the pre-processed video signal to a coding device. The coding device performs coding on the pre-processed video signal, and outputs a coded coded video signal. The first neural network model is trained using a neural network video coding unit including a trained second neural network model. The second neural network model is trained using a video signal of a first predicted image generated when the video coding unit codes the second video signal, and a video signal of a second predicted image generated when the neural network video coding unit codes the second video signal. Effect of the Invention

[0012] According to the present disclosure, it is possible to provide an encoding device, an encoding program, a pre-processing device, and a pre-processing program that are capable of improving not only the subjective image quality but also the objective image quality. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an encoding device at the time of inference according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of an encoding device during learning according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of a video encoding unit according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a picture reference structure according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of the configuration of a neural network video encoding unit according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of an encoding device during learning according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of an operation during learning according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of an operation during inference according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the configuration of an encoding system at the time of inference according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of an encoding system during learning according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of the configuration of a DVC encoder according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of the configuration of a VVC encoder according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [First embodiment] (Encoding method using neural networks) As described above, encoding methods using neural networks have been proposed. Here, the encoding method according to Non-Patent Document 1 and the encoding method according to Non-Patent Document 2 will be described.

[0015] (Encoding method according to Non-Patent Document 1) In Non-Patent Document 1, the encoding method using a neural network is called DVC (Deep Video Compression). Fig. 11 is a diagram showing an example of the configuration of a DVC encoder according to the first embodiment. The procedure of the encoding process in the DVC encoder will be described below with reference to Fig. 11.

[0016] Step N1 (motion detection): The current frame and the decoded past frame are input, and an optical flow value (i.e., motion information) is obtained using a CNN (optical flow network). Then, the motion information is compressed and decoded using a CNN (MV encoder network and MV decoder network) or the like to obtain quantized motion information.

[0017] Step N2 (motion compensation): Using the quantized motion information and the past frame as input, a predicted frame is obtained using a CNN (motion compensation network).

[0018] Step N3-N4 (transformation, quantization, and inverse transformation): A residual frame between the current frame and the predicted frame is output using a CNN (residual encoder network), and the output is quantized to obtain a quantized value. A reconstructed residual frame is obtained from the quantized value using a CNN (residual decoder network).

[0019] Step N5 (entropy coding): The quantized motion information obtained in step N1 and the quantized values ​​obtained in steps N3-N4 are used as inputs to obtain coded bits using a CNN (bitrate estimation network).

[0020] Step N6 (frame reconstruction): The predicted frame obtained in step N2 is added to the reconstructed residual frame obtained in steps N3-N4 to obtain a reconstructed frame (decoded frame).

[0021] The above is the encoding procedure by DVC. In DVC, during learning, a loss function (λD+R) is used using the distortion D between the current frame and the decoded frame obtained in step N6 and the number of bits used for encoding (input in step N6) R. In DVC, it is possible to construct an end-to-end optimal neural network encoding model by using the loss function (λD+R) to find parameters (or solve an optimization problem) that minimize the number of bits R and minimize the distortion D.

[0022] (Encoding method according to Non-Patent Document 2) Also, as described above, an encoding method (for example, Non-Patent Document 2) has been proposed that performs pre-processing before encoding. In Non-Patent Document 2, pre-processing is called DPP (Deep Perceptual Preprocessing). In an encoding method using DPP, during learning, the main components of encoding are virtualized, DPP is trained, and DPP is optimized using a loss function. Then, during inference, pre-processing is performed on an input image using the optimized DPP, and encoding processing is performed using an existing codec on the image after pre-processing.

[0023] In the encoding method using DPP, even if the encoding process is performed with any codec at any bit rate, it is possible to obtain an optimized DPP that balances perception and distortion using a loss function. Therefore, even if an image encoded with the encoding method using DPP is decoded, the visual quality can be improved compared to when DPP is not performed. In other words, the encoding method using DPP can improve the encoding efficiency (better image quality even with the same amount of data) compared to encoding methods that do not use DPP.

[0024] The above is an example of an encoding method using a neural network.

[0025] Fig. 12 is a diagram showing an example of the configuration of a VVC encoder according to the first embodiment. As described above, in DVC, a neural network is used to perform encoding that mimics the structure of a video encoding standard such as HEVC or VVC. For example, when the DVC encoder shown in Fig. 11 is compared with the VVC encoder shown in Fig. 12, the two encoders t For the decoded image x t-1 ^ Predicted footage from x t - Generate input video x t And prediction video x t - The residual signal r t r, which is a result of transforming and quantizing t ^ and predicted image x t - The sum of x and x is t The same is true for the flow that generates

[0026] In the above-mentioned Non-Patent Document 3, a neural network called an auxiliary codec network is used as a neural network that imitates a video coding standard. In addition, Non-Patent Document 3 discusses training the auxiliary codec network so that the difference between the output video from the auxiliary codec network and the output video from the video coding standard is minimized.

[0027] However, Non-Patent Document 3 does not discuss training the auxiliary codec network using intermediate predicted images.

[0028] As described above, although the DVC encoder and the VVC encoder are the same in that they use predicted images, the technology shown in Non-Patent Document 3 does not take into consideration the use of predicted images, so the auxiliary codec network cannot be imitated by the video encoding standard, and a certain level of accuracy cannot be obtained. Therefore, the technology shown in Non-Patent Document 3 may not improve the performance of the neural network model as a whole.

[0029] Therefore, an object of the first embodiment is to provide a video coding model and a pre-processing model having a certain level of accuracy or higher.

[0030] As described above, the video coding model is a neural network model used for coding a video signal. DVC, disclosed in Non-Patent Document 1, is an example of a video coding model.

[0031] The pre-processing model is a neural network model used in pre-processing. DPP, disclosed in Non-Patent Document 2, is an example of a pre-processing model.

[0032] Hereinafter, the first embodiment will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0033] (1. Example of the configuration of the encoding device during inference) Next, an example of the configuration of an encoding device will be described.

[0034] FIG. 1 is a diagram illustrating an example of the configuration of an encoding device according to the first embodiment.

[0035] As shown in Fig. 1, the encoding device 100 includes a pre-processing unit 110 and a video encoding unit 120. Here, the encoding device 100 shown in Fig. 1 represents an example of the configuration of the encoding device 100 at the time of inference.

[0036] That is, the pre-processing unit 110 includes a neural network model (for example, a first neural network model) that has already been trained, and pre-processing is performed using the trained model. The neural network model may be referred to as a "pre-processing model." The pre-processing may be, for example, a process of upscaling the resolution of the input video, or a process of downscaling the resolution of the input video. Alternatively, the pre-processing may be a process that does not change the resolution. The pre-processing unit 110 may perform pre-processing using any picture reference structure, but in the first embodiment, for example, the pre-processing unit 110 will be described as using the MPEG2 picture reference structure (GOP (Group Of Picture) structure with 15 intra periods and 2 consecutive frames) as shown in FIG. 4. In this way, the pre-processing unit 110 performs pre-processing using a trained neural network, and outputs the video signal of the pre-processing unit.

[0037] The video encoding unit 120 performs encoding processing on the pre-processed video signal and outputs the encoded encoded video signal. The encoding method in the video encoding unit 120 may be any method of the video encoding standard such as HEVC or VVC, but the first embodiment will be described assuming that MPEG2 is used. During inference, the video encoding unit 120 performs encoding processing using the video encoding standard used during learning. Furthermore, during inference, the video encoding unit 120 performs encoding processing using the same picture reference structure as the picture reference structure used during learning (e.g., FIG. 4).

[0038] In the first embodiment, first, the neural network video encoding unit is trained, and then the trained neural network video encoding unit is used to train the pre-processing unit 110. Here, it is also possible to train the pre-processing unit 110 using the video encoding unit 120. However, in the first embodiment, the pre-processing unit 110 is not trained using the video encoding unit 120, but is trained using the trained neural network video encoding unit. The reason for this is, for example, as follows.

[0039] That is, in the video encoding unit 120, a quantization process is performed on a signal after conversion process (here, DCT conversion process), but a part of the signal components may be deleted by rounding the quantized signal. Since such rounding is not differentiable, the error backpropagation method, which is a learning method for neural networks, cannot be used. On the other hand, in the neural network video encoding unit, such rounding is not performed, but the quantization process and the like are reproduced using a neural network by replacing the rounding with the addition of random noise. Therefore, it is expected that the learning result for the pre-processing unit 110 will be better when the neural network video encoding unit is used than when the video encoding unit 120 is used.

[0040] In the first embodiment, a neural network video encoding unit that performs encoding using a neural network model is used in training of the pre-processing unit 110, instead of a video encoding unit that performs encoding using a video encoding standard. The neural network model may be referred to as a "video encoding model."

[0041] In the following, as a configuration example of the encoding device 100 during training, (2.1) a configuration example of the encoding device 100 when training is performed on the neural network video encoding unit will be described. Next, (2.2) a configuration example of the encoding device 100 when training is performed on the pre-processing unit 110 using a trained neural network video encoding unit will be described.

[0042] In the following, there may be cases where the terms "neural network model" and "neural network" are used without distinction. For example, there may be cases where "performing pre-processing using a neural network model" and "performing pre-processing using a neural network" are used without distinction.

[0043] In addition, the pre-processing unit 110 has a neural network model (e.g., a first neural network model), and learning or inference is performed on the neural network model, but there are cases where the terms "learning and inference on the neural network model included in the pre-processing unit" and "learning and inference on the pre-processing unit 110" are used interchangeably.

[0044] Furthermore, the neural network video encoding unit 130 also has a neural network model (e.g., a second neural network model), and learning or inference is performed on the neural network model. However, there are cases where the terms "learning and inference on the neural network model included in the neural network video encoding unit 130" and "learning and inference on the neural network video encoding unit 130" are used interchangeably.

[0045] Furthermore, there are cases where "processing on video" and "processing on video signals" are used without distinction. For example, there are cases where "encoding processing on input video" and "encoding processing on video signals of input video" are used without distinction.

[0046] (2. Example of the configuration of the encoding device during learning)

[0047] (2.1 Example of the configuration of the encoding device when learning is performed in the neural network video encoding unit) Fig. 2 is a diagram illustrating an example of the configuration of the encoding device according to the first embodiment. Fig. 2 illustrates an example of the configuration of the encoding device 100 when training is performed on the neural network video encoding unit 130.

[0048] As shown in FIG. 2, the encoding device 100 includes a video encoding unit 120, a neural network video encoding unit 130, and an error determining unit 140 (eg, a first error determining unit).

[0049] The video encoding unit 120 may be the same as the video encoding unit 120 (FIG. 1) used during inference. The video encoding unit 120 performs encoding processing on a video signal of an input video (for example, a second video signal) using a picture reference structure, and outputs an encoded encoded video signal. The encoding itself may use a method according to a video encoding standard, and in the first embodiment, it is assumed that MPEG2 is used as described above.

[0050] Fig. 3 is a diagram illustrating an example of the configuration of the video encoding unit 120 according to the first embodiment. As illustrated in Fig. 3, the video encoding unit 120 has an intra encoding unit 121 and an inter encoding unit 122. The video encoding unit 120 performs encoding processing on the video signal of each video frame in accordance with the picture reference structure illustrated in Fig. 4.

[0051] That is, the first video frame in the GOP is treated as an I frame, and the intra coding unit 121 performs intra coding processing. The intra coding unit 121 outputs a video signal of the coded I frame. The second to sixteenth video frames in the GOP are input to the inter coding unit 122 as B frames or P frames according to the picture reference structure shown in FIG. 4, and inter coding processing is performed sequentially. The inter coding unit 122 outputs a video signal of the coded B frame or a video signal of the coded P frame. The video signals of the coded I frame, the coded B frame, and the coded P frame are output as video signals of coded video according to the temporal order shown in the picture reference structure shown in FIG. 4.

[0052] In the first embodiment, the inter-encoding unit 122 generates a video signal of a predicted image (for example, a first predicted image) and outputs the video signal. Although Fig. 12 shows an example of a VVC encoder, even in the case of an MPEG2 encoder, a predicted image can be generated by performing motion compensation on a decoded image, and the video signal of the predicted image can be output from the inter-encoding unit 122.

[0053] Returning to FIG. 2, the video encoding unit 120 outputs the video signal of the encoded video and the video signal of the predicted image to the error determining unit 140.

[0054] The neural network video encoding unit 130 performs intra-coding and inter-coding on an input video signal (e.g., a second video signal) using a neural network model (e.g., a second neural network model or a video encoding model), and outputs a neural network-encoded video signal (hereinafter, sometimes referred to as a “neural network-encoded video signal”). The neural network video encoding unit 130 outputs the neural network-encoded video signal to the error determining unit 140.

[0055] Fig. 5 is a diagram illustrating an example of the configuration of the neural network video encoding unit 130. As illustrated in Fig. 5, the neural network video encoding unit 130 includes a neural network intra encoding unit 131, a neural network inter encoding unit 132, and a neural network code amount calculation unit 133.

[0056] The neural network video encoding unit 130 may perform the encoding process using any picture reference structure, but in the first embodiment, the encoding process will be described as being performed using the same picture reference structure (e.g., FIG. 4) as the picture reference structure used by the video encoding unit 120.

[0057] In this case, the first video frame in the GOP is input as an I-frame to the neural network intra-encoding unit 131. The neural network intra-encoding unit 131 performs intra-encoding processing on the I-frame using a neural network model. The neural network intra-encoding unit 131 outputs a video signal of the encoded I-frame.

[0058] While any neural network model may be used for the I-frame, the first embodiment will be described assuming that intra-encoding processing is performed by the encoding method described in the following Non-Patent Document 5. In Non-Patent Document 5, for example, input video is subjected to parameter conversion using a neural network including a convolution layer or the like to convert it into latent variables, and the latent variables are further subjected to parameter conversion using a neural network to convert them into latent variables in which standard deviations are summarized. Then, quantization and arithmetic conversion are performed on the latent variables.

[0059] Non-Patent Document 5: Johannes Balle, David Minnen, Saurabh Singh, Sung Jin Hwang, Nick Johnston: “Variational image compression with a scale hyperprior”, 2018; arXiv:1802.01436. Next, the second to sixteenth video frames in the GOP are input to the neural network inter-encoding unit 132 as B frames or P frames according to the picture reference structure shown in Fig. 4. In the neural network inter-encoding unit 132, an inter-encoding process using a neural network is sequentially performed on the video signal of each video frame. The neural network inter-encoding unit 132 outputs the video signal of the coded B frame and the video signal of the coded P frame. The video signal of each coded video frame is output as a video signal of neural network-coded video according to the temporal order shown in the picture reference structure shown in Fig. 4.

[0060] Note that, although any model may be used as the neural network model for the B and P frames, the first embodiment will be described assuming that DVC corresponding to bidirectional prediction and described in Non-Patent Document 1 is used. Alternatively, the neural network model for the B and P frames may be a neural network model performing motion compensation for multiple frames, as described in Non-Patent Document 6 below.

[0061] Non-patent document 6: M. Akin Yilmaz, A. Murat Tekalp: “End-to-End Rate-Distortion Optimized Learned Hierarchical Bi-Directional Video Compression”, 2021; arXiv:2112.09529. For example, when DVC described in Non-Patent Document 1 is used in the neural network inter-encoding unit 132, a predicted image can be output from a motion compensation network, which is a neural network model simulating motion compensation, as shown in Fig. 11. Therefore, it is possible to output a video signal of a predicted image (e.g., a second predicted image) from the neural network inter-encoding unit 132. The neural network inter-encoding unit 132 outputs the predicted image as a neural network predicted image.

[0062] 5, the neural network code amount calculation unit 133 receives an image signal of a neural network-encoded image and calculates the code amount of the image signal. The neural network code amount calculation unit 133 outputs the calculated code amount as the neural network code amount.

[0063] In this way, the neural network video encoding unit 130 outputs the video signal of the neural network encoded video, the neural network code amount, and the video signal of the neural network predicted image during its own learning.

[0064] As shown in Fig. 5, the neural network video encoding unit 130 has a neural network model used in the neural network intra-encoding unit 131 and a neural network model used in the neural network inter-encoding unit 132. Although the two neural network models are different, the neural network models included in the neural network video encoding unit 130 may be described as one neural network model below. That is, the neural network model used in the neural network intra-encoding unit 131 and the neural network model used in the neural network inter-encoding unit 132 may be combined and described as a neural network model in the neural network video encoding unit 130 (e.g., a second neural network model or a video encoding model).

[0065] Returning to FIG. 2, the error determination unit 140 performs error determination between the neural network encoded video signal, which is the output of the neural network video encoding unit 130 for the video signal of the input video (e.g., the second video signal), and the encoded video signal, which is the output of the video encoding unit 120 for the video signal of the input video, and optimizes the parameters of the neural network video encoding unit 130. That is, the error determination unit 140 optimizes the parameters of the neural network model (i.e., the video encoding model) used in the neural network video encoding unit 130, using the video signal of the predicted image (e.g., the first predicted image) generated when the video signal of the input video is encoded in the video encoding unit 120 and the video signal of the predicted image (e.g., the second predicted image) generated when the video signal of the input video is encoded in the neural network video encoding unit 130. Specifically, the error determination unit 140 optimizes the parameters of the video encoding model using the following loss function L1.

[0066]

number

number

number

[0067] The error determination unit 140 optimizes parameters of the video encoding model, for example, by using the video signal of the predicted video from the video encoding unit 120 and the video signal of the neural network predicted image from the neural network video encoding unit 130. Specifically, the error determination unit 140 performs optimization so as to minimize distortion between the video signal of the predicted video output from the video encoding unit 120 and the video signal of the neural network predicted image output from the neural network video encoding unit 130, distortion between the video signal of the encoded video output from the video encoding unit 120 and the video signal of the neural network encoded video output from the neural network video encoding unit 130, and the neural network code amount.

[0068] As described above, in the first embodiment, the video coding model is trained using the video signal of the predicted image generated when the video coding unit 120 encodes the video signal of the input video, and the video signal of the predicted image generated when the neural network video coding unit 130 encodes the video signal of the input video. Since the video coding model is trained using the predicted image, which is an intermediate image, it is possible to improve the accuracy of the video coding model's imitation of the video coding standard compared to the case where the video coding model is trained without using the intermediate image. Therefore, it is possible to make the accuracy of the video coding model, i.e., the neural network model used in the neural network video coding unit 130, higher than a certain level.

[0069] Next, in the encoding device 100, the neural network model of the pre-processing unit 110 (for example, the first neural network model or the pre-processing model) is trained using the neural network video encoding unit 130 having the trained model.

[0070] (2.2 Example of the configuration of the encoding device when learning is performed in the pre-processing unit 110) FIG. 6 is a diagram illustrating an example of the configuration of the encoding device 100 when learning is performed in the pre-processing unit 110. As shown in FIG.

[0071] As shown in FIG. 6, the encoding device 100 includes a pre-processing unit 110, a neural network video encoding unit 130, and an error determining unit 145 (eg, a second error determining unit).

[0072] The pre-processing unit 110 performs pre-processing on the video signal of the input video (for example, the third video signal) and outputs the pre-processed video signal after pre-processing. The pre-processing unit 110 may perform pre-processing using any picture reference structure, but in the first embodiment, the pre-processing is described as using the same picture reference structure (for example, FIG. 4) as the picture reference structure used in the video encoding unit 120. That is, the pre-processing unit 110 inputs the video signal of each of the 16 video frames, with 16 video frames as 1 GOP, and outputs the video signal of each of the 16 video frames. The pre-processing unit 110 performs pre-processing using a neural network model (for example, a first neural network model or a pre-processing model). The neural network model in the pre-processing unit 110 may be any model, but in the first embodiment, BasicVSR shown in the following non-patent document 7 is used. BasicVSR is a neural network model that can take time series correlation into account. BasicVSR is a Video Super Resolution (VSR) neural network model that expands the number of pixels, but by changing the final processing stage within the model, it is possible to output images with any resolution.

[0073] Non-patent document 7Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, Chen Change Loy: “BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond”, 2020; arXiv:2012.02181. The neural network video encoding unit 130 outputs a neural network-encoded video signal using a trained neural network model (or video encoding model) for the pre-processed video signal. The neural network video encoding unit 130 may perform encoding using any picture reference structure, but in the first embodiment, the neural network video encoding unit 130 uses the same picture reference structure (e.g., FIG. 4) as the picture reference structure used by the video encoding unit 120. That is, the neural network video encoding unit 130 performs encoding processing on each of the 16 pre-processed video frames using a trained neural network, and outputs each video frame of the 16 neural network-encoded video. The neural network video encoding unit 130 outputs the neural network-encoded video signal to the error determination unit 145. The neural network video encoding unit 130 also calculates the code amount of the neural network-encoded video signal, and outputs it to the error determination unit 145 as the neural network code amount. The calculation of the neural network code amount is performed using the neural network code amount calculation unit 133 (FIG. 5). An example of the configuration of the neural network video encoding unit 130 when learning is performed in the pre-processing unit 110 may be the example of the configuration shown in FIG.

[0074] The error determination unit 145 performs error determination between the neural network encoded video signal, which is the output of the neural network video encoding unit 130 for the pre-processed video signal, and the video signal of the input video input to the pre-processing unit 110, and optimizes parameters of the neural network (for example, a first neural network or a pre-processing model) used in the pre-processing unit 110. The error determination unit 145 may use any loss function, but in the first embodiment, the following loss function L2 is used to perform optimization.

[0075]

number

[0076] In this way, in the first embodiment, the pre-processing model is trained using a trained video coding model. Therefore, in the first embodiment, the pre-processing model is trained using a video coding model having a certain level of accuracy or higher, so that the trained pre-processing model can derive a trained model having a certain level of accuracy or higher, compared to a case where a video coding model trained without using a predicted image is used.

[0077] Therefore, in the first embodiment, it is possible to derive a video coding model and a pre-processing model having a certain level of accuracy or higher.

[0078] 6, when the pre-processing in the pre-processing unit 110 includes upscaling, the neural network video and the input video can be made to have the same resolution by performing downscaling on the neural network encoded video signal. When the pre-processing includes downscaling, the neural network video and the input video can be made to have the same resolution by performing upscaling on the neural network encoded video signal. The process of performing downscaling or upscaling on the neural network encoded video signal may be performed by the neural network video encoding unit 130 or the error determination unit 145, for example.

[0079] (Operation example according to the first embodiment) Next, an operation example according to the first embodiment will be described.

[0080] FIG. 7 is a diagram illustrating an example of an operation during learning according to the first embodiment.

[0081] As shown in FIG. 7, in step S10, the encoding device 100 starts the process.

[0082] In step S11, the neural network video encoding unit 130 receives a video signal of an input video, and outputs a video signal of a neural network-encoded video and a video signal of a neural network predicted image. Also in step S11, the video encoding unit 120 receives a video signal of an input video, and outputs a video signal of an encoded video. Furthermore, in step S11, the neural network video encoding unit 130 outputs a neural network code amount.

[0083] In step S12, the error determining unit 140 uses Equation 1 to determine an error between the neural-network-encoded video and the encoded video, and optimizes parameters for the neural network in the neural-network video encoding unit 130 using the predicted image. In the neural-network video encoding unit 130, a trained neural network model is derived.

[0084] In step S13, the pre-processing unit 110 performs pre-processing on the input video using a neural network model, and outputs the pre-processed video signal to the neural network video coding unit 130. The neural network video coding unit 130 performs coding on the pre-processed video signal using the trained neural network model, and outputs a neural network-coded video signal.

[0085] In step S14, the error determining unit 145 determines an error between the neural-network-encoded image and the input image, and optimizes parameters of the neural network model of the pre-processing unit 110. In the pre-processing unit 110, a trained neural network model is derived.

[0086] In step S15, the encoding device 100 ends the learning.

[0087] FIG. 8 is a diagram illustrating an example of an operation during inference according to the first embodiment.

[0088] As shown in FIG. 8, in step S20, the encoding device 100 starts inference.

[0089] In step S21, the pre-processing unit 110 performs pre-processing on the input video using a trained neural network model, and outputs a pre-processed video signal.

[0090] In step S22, the video encoding unit 120 performs encoding processing on the pre-processed video signal in accordance with a video encoding standard, and outputs an encoded video signal. The video encoding unit 120 may transmit the encoded video signal to the video decoding device as a bit stream.

[0091] In step S23, the encoding device 100 ends the inference.

[0092] [Second embodiment] Next, a second embodiment will be described.

[0093] In the first embodiment, an example has been described in which the encoding device 100 includes the pre-processing unit 110. In the second embodiment, an example will be described in which a pre-processing device having the pre-processing unit 110 is provided in the front stage of the encoding device 100.

[0094] FIG. 9 is a diagram illustrating an example of the configuration of an encoding system 10 according to the second embodiment.

[0095] 9, the encoding system 10 includes a pre-processing device 200 and an encoding device 100. The pre-processing device 200 includes a pre-processing unit 110. In addition, the encoding device 100 includes a video encoding unit 120.

[0096] The encoding system 10 shown in Fig. 9 shows an example of a configuration at the time of inference, similar to Fig. 1. That is, the pre-processing unit 110 performs pre-processing using an already trained neural network model (e.g., a first neural network model or a video encoding model). Specifically, the pre-processing unit 110 performs pre-processing on a video signal of an input video (e.g., a first video signal) using the trained neural network model, and outputs the pre-processed video signal to the encoding device 100.

[0097] As in the first embodiment, the encoding device 100 performs encoding processing on the pre-processed video signal using a video encoding standard, and outputs a video signal of encoded video.

[0098] FIG. 10 is a diagram illustrating an example of the configuration of the encoding system 10 during learning according to the second embodiment.

[0099] As shown in FIG. 10, during learning, the pre-processing device 200 has a pre-processing unit 110, and the encoding device 100 has a neural network video encoding unit 130 and an error determining unit 145.

[0100] As in the first embodiment (FIG. 6), the pre-processing unit 110 performs pre-processing on the video signal of the input video using a neural network model (e.g., a first neural network model or a pre-processing model), and outputs the pre-processed video signal.

[0101] The neural network video encoding unit 130 has a trained neural network model (e.g., a second neural network model or a video encoding model) that has been trained using a predicted image, as in the first embodiment. As in the first embodiment, the neural network video encoding unit 130 performs encoding processing on a video signal of a pre-processed video using the neural network model, and outputs a neural network-encoded video signal.

[0102] Similarly to the first embodiment, the error determining unit 145 also optimizes the parameters of the neural network used in the pre-processing unit 110 using the loss function L2 shown in Equation 2.

[0103] In the second embodiment as well, the learning of the pre-processing model is performed using a trained video coding model. Moreover, the learning of the video coding model is performed using a predicted image. Therefore, in the second embodiment as well, it is possible to derive a trained video coding model having a certain level of accuracy or higher, and further, by training the pre-processing model using the video coding model, it is possible to derive a trained pre-processing model having a certain level of accuracy or higher. Therefore, in the second embodiment as well, as in the first embodiment, it is possible to derive a video coding model and a pre-processing model having a certain level of accuracy or higher.

[0104] [Other embodiments] In the above-mentioned first and second embodiments, the intermediate image is described by taking the predicted image as an example, but the intermediate image used for learning the video coding model is not limited to the predicted image. For example, a decoded image obtained by adding a predicted image to a residual signal after inverse quantization may be used as the intermediate image used for learning the video coding model. In this case, it is possible to implement by replacing the predicted image generated by the video coding unit 120 described in the first and second embodiments with the decoded image generated by the video coding unit 120. Also, it is possible to implement by replacing the predicted image generated by the neural network video coding unit 130 described in the first and second embodiments with the decoded image generated by the neural network video coding unit 130. The intermediate image used for learning the video coding model may be, for example, an image generated between an input image and a coded image in a video coding standard.

[0105] Furthermore, in the first and second embodiments, an example has been described in which the pre-processing unit 110 is trained after the neural network video encoding unit 130 is trained, but after the pre-processing unit 110 is trained, the neural network video encoding unit 130 may be trained again, and the pre-processing unit 110 may then be trained again. In other words, the neural network video encoding unit 130 may be trained and then the pre-processing unit 110 may be trained repeatedly.

[0106] Also, a program may be provided that causes a computer to execute each process performed by the above-mentioned devices (encoding device 100 and pre-processing device 200). The program may be recorded in a computer-readable medium. Using the computer-readable medium, it is possible to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Also, circuits that execute each process performed by the above-mentioned devices (encoding device 100 and pre-processing device 200) may be integrated, and the device may be configured as a semiconductor integrated circuit (chip set, SoC).

[0107] Although the embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes can be made without departing from the scope of the invention. In addition, each operation example can be combined without any contradiction. [Explanation of symbols]

[0108] 100: Encoding device 110: Pre-processing unit 120: Video encoding unit 121: Intra-coding unit 122: Inter-encoding unit 130: Neural network video encoding unit 131: Neural network intra coding unit 132: Neural network inter-encoding unit 133: Neural network code amount calculation unit 140, 145: Error determination unit 200: Pre-processing device

Claims

1. a pre-processing unit that performs pre-processing on the first video signal by using the trained first neural network model and outputs the pre-processed video signal; a video encoding unit that performs an encoding process on the pre-processed video signal and outputs an encoded video signal, the training of the first neural network model is performed using a neural network video encoding unit including a trained second neural network model; The second neural network model is trained using a video signal of a first predicted image generated when the video encoding unit encodes the second video signal, and a video signal of a second predicted image generated when the neural network video encoding unit encodes the second video signal. Encoding device.

2. The neural network video encoding unit performs intra-encoding and inter-encoding on the second video signal using the second neural network model, and outputs a neural network-encoded video signal.

2. The encoding device according to claim 1.

3. The image processing apparatus further includes a first error determination unit that optimizes parameters of the second neural network model by using a video signal of the first predicted image and a video signal of the second predicted image when learning the second neural network model.

3. The encoding device according to claim 2.

4. The first error determining unit optimizes parameters of the second neural network model so as to minimize a distortion between the video signal of the first predicted image and the video signal of the second predicted image, a distortion between the encoded video signal output from the video encoding unit and the neural network encoded video signal output from the neural network encoded video, and a code amount of the neural network encoded video signal.

4. The encoding device according to claim 3.

5. and a second error determination unit that performs error determination between a neural network-encoded video signal, which is an output of the neural network video encoding unit for the pre-processed video signal, and a third video signal input to the pre-processing unit, when the first neural network model is trained, and optimizes parameters of the first neural network model.

2. The encoding device according to claim 1.

6. An encoding program to be executed by a computer of an encoding device, performing pre-processing on the first video signal using the trained first neural network model, and outputting the pre-processed video signal; a process of causing a computer to execute a process of encoding the pre-processed video signal and outputting the encoded video signal; the training of the first neural network model is performed using a neural network video encoding unit including a trained second neural network model; The second neural network model is trained using a video signal of a first predicted image generated when the video encoding unit encodes the second video signal, and a video signal of a second predicted image generated when the neural network video encoding unit encodes the second video signal. Encoding program.

7. A pre-processing device, a pre-processing unit that performs pre-processing on the first video signal by using the trained first neural network model and outputs the pre-processed video signal to the encoding device; the encoding device performs an encoding process on the pre-processed video signal, and outputs the encoded video signal; the training of the first neural network model is performed using a neural network video encoding unit including a trained second neural network model; The second neural network model is trained using a video signal of a first predicted image generated when the video encoding unit encodes the second video signal, and a video signal of a second predicted image generated when the neural network video encoding unit encodes the second video signal. Pre-processing device.

8. A pre-processing program executed by a computer of a pre-processing device, performing pre-processing on the first video signal using the trained first neural network model, and outputting the pre-processed video signal to an encoding device; the encoding device performs an encoding process on the pre-processed video signal, and outputs the encoded video signal; the training of the first neural network model is performed using a neural network video encoding unit including a trained second neural network model; The second neural network model is trained using a video signal of a first predicted image generated when the video encoding unit encodes the second video signal, and a video signal of a second predicted image generated when the neural network video encoding unit encodes the second video signal. Pre-processing program.

Images