Encoding device, encoding program, pre-processing device, and pre-processing program

Abstract

To provide encoding device and program, and pre-processing device and program, each having a video encoding model and a pre-processing model with a certain level of accuracy or more level of accuracy.SOLUTION: An encoding device 100 using a neural network includes a video encoding unit 120 that performs encoding processing on a video signal of an input video using a picture reference structure, and outputs the processed encoded video signal, a neural network video encoding unit 130, and an error determination unit 140. The neural network video encoding unit includes a trained second neural network model and performs learning on the first neural network model. Learning on the second neural network model is performed on the basis of a code amount of the encoded video signal encoded by the video encoding unit for the second video signal and a code amount of the neural network encoded video signal encoded by the neural network video encoding unit for the second video signal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an encoding device, an encoding program, a pre-processing device, and a pre-processing program. [Background technology]

[0002] In recent years, coding methods using neural networks have been attracting attention.

[0003] For example, there is an encoding method that uses a CNN (Convolutional Neural Network) to perform a motion detection process that detects a motion vector between a current image and a past image, a motion compensation process that generates a predicted image from the motion vector, and a conversion process that converts (e.g., DCT conversion) a difference image between an original image and a predicted image (e.g., Non-Patent Document 1). By optimizing such an encoding method end-to-end, it is possible to obtain encoding efficiency equivalent to that of a conventional encoding method such as HEVC (High Efficiency Video Coding).

[0004] In addition to the coding method using a neural network, there is also a coding method that performs preprocessing before coding (for example, Non-Patent Documents 2 and 3). Regarding the coding method using preprocessing, it has been reported that the coding efficiency is improved when a subjective image quality index based on MS-SSIM (Multi-Scale Structural Similarity) or VMAF (Video Multimethod Assessment Fusion) is used, compared to conventional coding methods such as H.264.

[0005] Furthermore, there is a technology that applies pre-processing and encoding processing to a machine vision task of automatically detecting and classifying objects in an image (for example, Non-Patent Document 4). This technology also aims to jointly optimize pre-processing, encoding processing, and machine vision tasks, which are configured using a neural network model, in an end-to-end manner during learning.

Prior technical literature

Non-licensed literature

[0006] [Non-licensed document 1] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai and Z. Gao, "DVC: An End-To-End Deep Video Compression Framework," 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 April 7, pp. 10998-11007.

Non-licensed Document 2

Non-licensed Document 4

[0007] An object of the present disclosure is to provide an encoding device, an encoding program, a pre-processing device, and a pre-processing program that have a video encoding model and a pre-processing model with a certain level of accuracy or higher. [Means for solving the problem]

[0008] The encoding device according to the first aspect includes a pre-processing unit that performs pre-processing on a first video signal using a trained first neural network model and outputs a pre-processed video signal. The encoding device also includes a video encoding unit that performs encoding on the pre-processed video signal and outputs an encoded encoded video signal. Here, the first neural network model is trained using a neural network video encoding unit including a trained second neural network model. The second neural network model is trained based on the code amount of the encoded video signal encoded by the video encoding unit for the second video signal and the code amount of the neural network encoded video signal encoded by the neural network video encoding unit for the second video signal.

[0009] A coding program according to a second aspect is a coding program executed by a computer of a coding device. The coding program causes the computer to execute a process of performing pre-processing on a first video signal using a trained first neural network model, and outputting the pre-processed video signal. The coding program also causes the computer to execute a process of encoding the pre-processed video signal, and outputting the encoded encoded video signal. The first neural network model is trained using a neural network video encoding unit including a trained second neural network model. The second neural network model is trained based on the code amount of the encoded video signal encoded by the video encoding unit for the second video signal, and the code amount of the neural network encoded video signal encoded by the neural network video encoding unit for the second video signal.

[0010] A pre-processing device according to a third aspect has a pre-processing unit that performs pre-processing on a first video signal using a trained first neural network model and outputs the pre-processed video signal to an encoding device. The encoding device performs encoding processing on the pre-processed video signal and outputs an encoded encoded video signal. The first neural network model is trained using a neural network video encoding unit including a trained second neural network model. Furthermore, the second neural network model is trained based on the code amount of the encoded video signal encoded by the video encoding unit for the second video signal and the code amount of the neural network encoded video signal encoded by the neural network video encoding unit for the second video signal.

[0011] A pre-processing program according to a fourth aspect is a pre-processing program executed by a computer of a pre-processing device. The pre-processing program causes the computer to execute a process of performing pre-processing on a first video signal using a trained first neural network model, and outputting the pre-processed video signal to an encoding device. The encoding device performs encoding on the pre-processed video signal, and outputs an encoded encoded video signal. The first neural network model is trained using a neural network video encoding unit including a trained second neural network model. Furthermore, the second neural network model is trained based on the code amount of the encoded video signal encoded by the video encoding unit for the second video signal, and the code amount of the neural network encoded video signal encoded by the neural network video encoding unit for the second video signal. Effect of the Invention

[0012] According to the present disclosure, it is possible to provide an encoding device, an encoding program, a pre-processing device, and a pre-processing program that have a video encoding model and a pre-processing model with a certain level of accuracy or higher. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an encoding device at the time of inference according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of an encoding device during learning according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of a video encoding unit according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a picture reference structure according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of the configuration of a neural network video encoding unit according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of an encoding device during learning according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of an operation during learning according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of an operation during inference according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the configuration of an encoding system at the time of inference according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of an encoding system during learning according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [First embodiment] (Encoding method using neural networks) As described above, encoding methods using neural networks have been proposed. Here, the encoding method according to Non-Patent Document 1 and the encoding method according to Non-Patent Document 2 will be described.

[0015] (Encoding method according to Non-Patent Document 1) In Non-Patent Document 1, the coding method using a neural network is called DVC (Deep Video Compression). In DVC, coding processing is performed according to the following procedure.

[0016] Step N1 (motion detection): The current frame and the decoded past frame are input, and an optical flow value (i.e., motion information) is obtained using a CNN (optical flow network). Then, the motion information is compressed and decoded using a CNN (MV encoder network and MV decoder network) or the like to obtain quantized motion information.

[0017] Step N2 (motion compensation): Using the quantized motion information and the past frame as input, a predicted frame is obtained using a CNN (motion compensation network).

[0018] Step N3-N4 (transformation, quantization, and inverse transformation): A residual frame between the current frame and the predicted frame is output using a CNN (residual encoder network), and the output is quantized to obtain a quantized value. A reconstructed residual frame is obtained from the quantized value using a CNN (residual decoder network).

[0019] Step N5 (entropy coding): The quantized motion information obtained in step N1 and the quantized values ​​obtained in steps N3-N4 are used as inputs to obtain coded bits using a CNN (bitrate estimation network).

[0020] Step N6 (frame reconstruction): The predicted frame obtained in step N2 is added to the reconstructed residual frame obtained in steps N3-N4 to obtain a reconstructed frame (decoded frame).

[0021] The above is the encoding procedure by DVC. In DVC, during learning, a loss function (λD+R) is used using the distortion D between the current frame and the decoded frame obtained in step N6 and the number of bits used for encoding (input in step N6) R. In DVC, it is possible to construct an end-to-end optimal neural network encoding model by using the loss function (λD+R) to find parameters (or solve an optimization problem) that minimize the number of bits R and minimize the distortion D.

[0022] (Encoding method according to Non-Patent Document 2) Also, as described above, an encoding method (for example, Non-Patent Document 2) has been proposed that performs pre-processing before encoding. In Non-Patent Document 2, pre-processing is called DPP (Deep Perceptual Preprocessing). In an encoding method using DPP, during learning, the main components of encoding are virtualized, DPP is trained, and DPP is optimized using a loss function. Then, during inference, pre-processing is performed on an input image using the optimized DPP, and encoding processing is performed using an existing codec on the image after pre-processing.

[0023] In the encoding method using DPP, even if the encoding process is performed with any codec at any bit rate, it is possible to obtain an optimized DPP that balances perception and distortion using a loss function. Therefore, even if an image encoded with the encoding method using DPP is decoded, the visual quality can be improved compared to when DPP is not performed. In other words, the encoding method using DPP can improve the encoding efficiency (better image quality even with the same amount of data) compared to encoding methods that do not use DPP.

[0024] The above is an example of an encoding method using a neural network.

[0025] As mentioned above, there is a technology (Non-Patent Document 4) that applies pre-processing and encoding processes configured with neural networks to machine vision tasks. Non-Patent Document 4 shows that the accuracy of classification and detection is improved when the pre-processing is used compared to when the pre-processing is not used. In addition, Non-Patent Document 4 also shows that the following loss function L p is used to optimize the neural network used in the encoding process (referred to as a "proxy network" in Non-Patent Document 4).

[0026]

number

[0027] In general, it is said that since the video coding standard and the video coding model that performs coding using a neural network model such as DPP (proxy network model in Non-Patent Document 4) have different coding performances, it is difficult to perform learning so that the code amount values ​​generated in the two coding processes approach each other. In particular, in the video coding standard, coding is performed sequentially according to the picture reference structure of the encoder, and a quantization parameter QP (Quantization Parameter) is set for each picture, so that the code amount generated by coding may also differ for each picture.

[0028] Therefore, in the first embodiment, the purpose is to derive a video coding model and a pre-processing model having a certain level of accuracy or higher. By performing inference using a video coding model and a pre-processing model having a certain level of accuracy or higher, it is possible to improve the image quality of the coded video. In addition, even when a machine vision task is performed using the coded video, it is possible to perform detection and classification with a certain level of accuracy or higher.

[0029] As described above, the video coding model is a neural network model used for coding a video signal. DVC, disclosed in Non-Patent Document 1, is an example of a video coding model.

[0030] The pre-processing model is a neural network model used in pre-processing. DPP, disclosed in Non-Patent Document 2, is an example of a pre-processing model.

[0031] Hereinafter, the first embodiment will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0032] (1. Example of the configuration of the encoding device during inference) Next, an example of the configuration of an encoding device will be described.

[0033] FIG. 1 is a diagram illustrating an example of the configuration of an encoding device according to the first embodiment.

[0034] As shown in Fig. 1, the encoding device 100 includes a pre-processing unit 110 and a video encoding unit 120. Here, the encoding device 100 shown in Fig. 1 represents an example of the configuration of the encoding device 100 at the time of inference.

[0035] That is, the pre-processing unit 110 includes a neural network model (for example, a first neural network model) that has already been trained, and pre-processing is performed using the trained model. The neural network model may be referred to as a "pre-processing model." The pre-processing may be, for example, a process of upscaling the resolution of the input video, or a process of downscaling the resolution of the input video. Alternatively, the pre-processing may be a process that does not change the resolution. The pre-processing unit 110 may perform pre-processing using any picture reference structure, but in the first embodiment, for example, the pre-processing unit 110 will be described as using the MPEG2 picture reference structure (GOP (Group Of Picture) structure with 15 intra periods and 2 consecutive frames) as shown in FIG. 4. In this way, the pre-processing unit 110 performs pre-processing using a trained neural network, and outputs the video signal of the pre-processing unit.

[0036] The video encoding unit 120 performs encoding processing on the pre-processed video signal and outputs the encoded encoded video signal. The encoding method in the video encoding unit 120 may be any method of the video encoding standard such as HEVC or VVC, but the first embodiment will be described assuming that MPEG2 is used. During inference, the video encoding unit 120 performs encoding processing using the video encoding standard used during learning. Furthermore, during inference, the video encoding unit 120 performs encoding processing using the same picture reference structure as the picture reference structure used during learning.

[0037] In the first embodiment, first, the neural network video encoding unit is trained, and then the trained neural network video encoding unit is used to train the pre-processing unit 110. Here, it is also possible to train the pre-processing unit 110 using the video encoding unit 120. However, in the first embodiment, the pre-processing unit 110 is not trained using the video encoding unit 120, but is trained using the trained neural network video encoding unit. The reason for this is, for example, as follows.

[0038] That is, in the video encoding unit 120, a quantization process is performed on a signal after conversion process (here, DCT conversion process), but a part of the signal components may be deleted by rounding the quantized signal. Since such rounding is not differentiable, the error backpropagation method, which is a learning method for neural networks, cannot be used. On the other hand, in the neural network video encoding unit, such rounding is not performed, but the quantization process and the like are reproduced using a neural network by replacing the rounding with the addition of random noise. Therefore, it is expected that the learning result for the pre-processing unit 110 will be better when the neural network video encoding unit is used than when the video encoding unit 120 is used.

[0039] In the first embodiment, a neural network video encoding unit that performs encoding using a neural network model is used in training of the pre-processing unit 110, instead of a video encoding unit that performs encoding using a video encoding standard. The neural network model may be referred to as a "video encoding model."

[0040] In the following, as a configuration example of the encoding device 100 during training, (2.1) a configuration example of the encoding device 100 when training is performed on the neural network video encoding unit will be described. Next, (2.2) a configuration example of the encoding device 100 when training is performed on the pre-processing unit 110 using a trained neural network video encoding unit will be described.

[0041] In the following, there may be cases where the terms "neural network model" and "neural network" are used without distinction. For example, there may be cases where "performing pre-processing using a neural network model" and "performing pre-processing using a neural network" are used without distinction.

[0042] In addition, the pre-processing unit 110 has a neural network model (e.g., a first neural network model), and learning or inference is performed on the neural network model, but there are cases where the terms "learning and inference on the neural network model included in the pre-processing unit" and "learning and inference on the pre-processing unit 110" are used interchangeably.

[0043] Furthermore, the neural network video encoding unit 130 also has a neural network model (e.g., a second neural network model), and learning or inference is performed on the neural network model. However, there are cases where the terms "learning and inference on the neural network model included in the neural network video encoding unit 130" and "learning and inference on the neural network video encoding unit 130" are used interchangeably.

[0044] Furthermore, there are cases where "processing on video" and "processing on video signals" are used without distinction. For example, there are cases where "encoding processing on input video" and "encoding processing on video signals of input video" are used without distinction.

[0045] (2. Example of the configuration of the encoding device during learning)

[0046] (2.1 Example of the configuration of the encoding device when learning is performed in the neural network video encoding unit) Fig. 2 is a diagram illustrating an example of the configuration of the encoding device according to the first embodiment. Fig. 2 illustrates an example of the configuration of the encoding device 100 when training is performed on the neural network video encoding unit 130.

[0047] As shown in FIG. 2, the encoding device 100 includes a video encoding unit 120, a neural network video encoding unit 130, and an error determining unit 140 (eg, a first error determining unit).

[0048] The video encoding unit 120 may be the same as the video encoding unit 120 (FIG. 1) used for inference. The video encoding unit 120 performs encoding processing on the video signal of the input video by using a picture reference structure, and outputs the encoded encoded video signal. The encoding itself may be performed using a method according to a video encoding standard, and in the first embodiment, it is assumed that MPEG2 is used as described above.

[0049] Fig. 3 is a diagram illustrating an example of the configuration of the video encoding unit 120 according to the first embodiment. As illustrated in Fig. 3, the video encoding unit 120 includes an intra encoding unit 121, an inter encoding unit 122, and an encoding code amount calculation unit 123. The video encoding unit 120 performs encoding processing on the video signal of each video frame in accordance with the picture reference structure illustrated in Fig. 4.

[0050] That is, the first video frame in the GOP is treated as an I frame, and the intra coding unit 121 performs intra coding processing. The intra coding unit 121 outputs a video signal of the coded I frame. The second to sixteenth video frames in the GOP are input to the inter coding unit 122 as B frames or P frames according to the picture reference structure shown in FIG. 4, and inter coding processing is performed sequentially. The inter coding unit 122 outputs a video signal of the coded B frame or a video signal of the coded P frame. The video signals of the coded I frame, the coded B frame, and the coded P frame are output as video signals of coded video according to the temporal order shown in the picture reference structure shown in FIG. 4.

[0051] The coding code amount calculation unit 123 calculates the amount of codes of the video signal of the coded video. The coding code amount calculation unit 123 outputs the calculated amount of codes as the amount of coded codes.

[0052] Returning to FIG. 2, the video encoding unit 120 outputs the video signal of the encoded video and the amount of encoded code to the error determining unit 140.

[0053] The neural network video encoding unit 130 performs intra-coding and inter-coding on an input video signal (e.g., a second video signal) using a neural network model (e.g., a second neural network model or a video encoding model), and outputs a neural network-encoded video signal (hereinafter, sometimes referred to as a “neural network-encoded video signal”). The neural network video encoding unit 130 outputs the neural network-encoded video signal to the error determining unit 140.

[0054] Fig. 5 is a diagram illustrating an example of the configuration of the neural network video encoding unit 130. As illustrated in Fig. 5, the neural network video encoding unit 130 includes a neural network intra encoding unit 131, a neural network inter encoding unit 132, and a neural network code amount calculation unit 133.

[0055] The neural network video encoding unit 130 may perform the encoding process using any picture reference structure, but in the first embodiment, the encoding process will be described as being performed using the same picture reference structure (e.g., FIG. 4) as the picture reference structure used by the video encoding unit 120.

[0056] In this case, the first video frame in the GOP is input as an I-frame to the neural network intra-encoding unit 131. The neural network intra-encoding unit 131 performs intra-encoding processing on the I-frame using a neural network model. The neural network intra-encoding unit 131 outputs a video signal of the encoded I-frame.

[0057] While any neural network model may be used for the I-frame, the first embodiment will be described assuming that intra-encoding processing is performed by the encoding method described in the following Non-Patent Document 5. In Non-Patent Document 5, for example, input video is subjected to parameter conversion using a neural network including a convolution layer or the like to convert it into latent variables, and the latent variables are further subjected to parameter conversion using a neural network to convert them into latent variables in which standard deviations are summarized. Then, quantization and arithmetic conversion are performed on the latent variables.

[0058] Non-Patent Document 5: Johannes Balle, David Minnen, Saurabh Singh, Sung Jin Hwang, Nick Johnston: “Variational image compression with a scale hyperprior”, 2018; arXiv:1802.01436. Next, the second to sixteenth video frames in the GOP are input to the neural network inter-encoding unit 132 as B frames or P frames according to the picture reference structure shown in Fig. 4. In the neural network inter-encoding unit 132, an inter-encoding process using a neural network is sequentially performed on the video signal of each video frame. The neural network inter-encoding unit 132 outputs the video signal of the coded B frame and the video signal of the coded P frame. The video signal of each coded video frame is output as a video signal of neural network-coded video according to the temporal order shown in the picture reference structure shown in Fig. 4.

[0059] Note that, although any model may be used as the neural network model for the B and P frames, the first embodiment will be described assuming that DVC corresponding to bidirectional prediction and described in Non-Patent Document 1 is used. Alternatively, the neural network model for the B and P frames may be a neural network model performing motion compensation for multiple frames, as described in Non-Patent Document 6 below.

[0060] Non-patent document 6: M. Akin Yilmaz, A. Murat Tekalp: “End-to-End Rate-Distortion Optimized Learned Hierarchical Bi-Directional Video Compression”, 2021; arXiv:2112.09529. The neural network code amount calculation unit 133 receives the video signal of the neural network encoded video, calculates the code amount of the video signal, and outputs the calculated code amount as the neural network code amount.

[0061] As shown in Fig. 5, the neural network video encoding unit 130 has a neural network model used in the neural network intra-encoding unit 131 and a neural network model used in the neural network inter-encoding unit 132. Although the two neural network models are different, the neural network models included in the neural network video encoding unit 130 may be described as one neural network model below. That is, the neural network model used in the neural network intra-encoding unit 131 and the neural network model used in the neural network inter-encoding unit 132 may be combined and described as a neural network model in the neural network video encoding unit 130 (e.g., a second neural network model or a video encoding model).

[0062] Returning to Fig. 2, the error determination unit 140 performs error determination between the neural network encoded video signal that is the output of the neural network video encoding unit 130 for the video signal of the input video (e.g., the second video signal) and the encoded video signal that is the output of the video encoding unit 120 for the video signal of the input video, and optimizes the parameters of the neural network video encoding unit 130. That is, the error determination unit 140 optimizes the parameters of the neural network model (i.e., the video encoding model) used in the neural network video encoding unit 130, using the code amount for each picture generated by the video encoding unit 120 and the code amount for each picture generated by the neural network video encoding unit 130. Specifically, the error determination unit 140 optimizes the parameters of the video encoding model using the following loss function L1.

[0063]

number

[0064]

number

[0065] As shown in Equation 2 and Equation 3, the error determining unit 140 optimizes the parameters of the video coding model by using the coding code amount and the neural network code amount when learning the video coding model. That is, by using the loss function L1 shown in Equation 2 and Equation 3, the error determining unit 140 optimizes the parameters of the video coding model not only to make the neural network coded video closer to the coded video, but also to make the ratio between the neural network code amount and the coding code amount closer to a constant value.

[0066] Next, in the encoding device 100, the neural network model of the pre-processing unit 110 (for example, the first neural network model or the pre-processing model) is trained using the neural network video encoding unit 130 having the trained model.

[0067] (2.2 Example of the configuration of the encoding device when learning is performed in the pre-processing unit 110) FIG. 6 is a diagram illustrating an example of the configuration of the encoding device 100 when learning is performed in the pre-processing unit 110. As shown in FIG.

[0068] As shown in FIG. 6, the encoding device 100 includes a pre-processing unit 110, a neural network video encoding unit 130, and an error determining unit 145 (eg, a second error determining unit).

[0069] The pre-processing unit 110 performs pre-processing on the video signal of the input video (for example, the third video signal) and outputs the pre-processed video signal after pre-processing. The pre-processing unit 110 may perform pre-processing using any picture reference structure, but in the first embodiment, the pre-processing is described as using the same picture reference structure (for example, FIG. 4) as the picture reference structure used in the video encoding unit 120. That is, the pre-processing unit 110 inputs the video signal of each of the 16 video frames, with 16 video frames as 1 GOP, and outputs the video signal of each of the 16 video frames. The pre-processing unit 110 performs pre-processing using a neural network model (for example, a first neural network model or a pre-processing model). The neural network model in the pre-processing unit 110 may be any model, but in the first embodiment, BasicVSR shown in the following non-patent document 7 is used. BasicVSR is a neural network model that can take time series correlation into account. BasicVSR is a Video Super Resolution (VSR) neural network model that expands the number of pixels, but by changing the final processing stage within the model, it is possible to output images with any resolution.

[0070] Non-patent document 7: Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, Chen Change Loy: “BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond”, 2020; arXiv:2012.02181. The neural network video encoding unit 130 outputs a neural network-encoded video signal using a trained neural network model (or video encoding model) for the pre-processed video signal. The neural network video encoding unit 130 may perform encoding using any picture reference structure, but in the first embodiment, the neural network video encoding unit 130 uses the same picture reference structure (e.g., FIG. 4) as the picture reference structure used by the video encoding unit 120. That is, the neural network video encoding unit 130 performs encoding processing on each of the 16 pre-processed video frames using a neural network, and outputs each of the 16 neural network-encoded video frames. The neural network video encoding unit 130 outputs the neural network-encoded video signal to the error determination unit 145. The neural network video encoding unit 130 also calculates the code amount of the neural network-encoded video signal, and outputs it to the error determination unit 145 as the neural network code amount. The calculation of the neural network code amount is performed using a neural network code amount calculation unit 133 (FIG. 5). An example of the configuration of the neural network video encoding unit 130 when learning is performed in the pre-processing unit 110 may be the example of the configuration shown in FIG.

[0071] The error determination unit 145 performs error determination between the neural network encoded video signal, which is the output of the neural network video encoding unit 130 for the pre-processed video signal, and the video signal of the input video input to the pre-processing unit 110, and optimizes parameters of the neural network (for example, a first neural network or a pre-processing model) used in the pre-processing unit 110. The error determination unit 145 may use any loss function, but in the first embodiment, the following loss function L2 is used to perform optimization.

[0072]

number

[0073] As described above, in the first embodiment, the neural network model (e.g., the first neural network model or the pre-processing model) in the pre-processing unit 110 is trained using the neural network video encoding unit 130 including a trained neural network model (e.g., the second neural network model or the video encoding model). Moreover, the neural network model included in the neural network video encoding unit 130 is trained based on the code amount (i.e., the coding code amount) of the encoded video signal encoded by the video encoding unit 120 for the video signal of the input video, and the code amount (i.e., the neural network code amount) of the neural network encoded video signal encoded by the neural network video encoding unit 130 for the video signal of the input video.

[0074] Therefore, in the first embodiment, it is possible to perform learning so that not only the neural network encoded video approaches the encoded video, but also the ratio of the two code amounts, the encoded code amount and the neural network code amount, approaches a constant value. As a result, for example, the accuracy of the video encoding model derived by learning can be made to be equal to or higher than a certain level. Then, since the pre-processing model is learned using such a video encoding model, it is possible to make the accuracy of the pre-processing model derived by learning equal to or higher than a certain level. By performing inference using the pre-processing model learned in this way, it is possible to improve the image quality of the encoded video to be equal to or higher than a certain level.

[0075] 6, when the pre-processing in the pre-processing unit 110 includes upscaling, the neural network video and the input video can be made to have the same resolution by performing downscaling on the neural network encoded video signal. When the pre-processing includes downscaling, the neural network video and the input video can be made to have the same resolution by performing upscaling on the neural network encoded video signal. The process of performing downscaling or upscaling on the neural network encoded video signal may be performed by the neural network video encoding unit 130 or the error determination unit 145, for example.

[0076] (Operation example according to the first embodiment) Next, an operation example according to the first embodiment will be described.

[0077] FIG. 7 is a diagram illustrating an example of an operation during learning according to the first embodiment.

[0078] As shown in FIG. 7, in step S10, the encoding device 100 starts the process.

[0079] In step S11, the neural network video encoding unit 130 receives the video signal of the input video and outputs a neural network-encoded video signal. Also in step S11, the video encoding unit 120 receives the video signal of the input video and outputs an encoded video signal.

[0080] In step S12, the neural network code amount calculation unit 133 calculates the code amount of the neural network encoded video signal. Also in step S12, the encoded code amount calculation unit 123 calculates the code amount of the encoded video signal.

[0081] In step S13, the error determining unit 140 optimizes parameters for the neural network (e.g., the second neural network model or the video encoding model) in the neural network video encoding unit 130 based on the neural network code amount and the encoding code amount using Equation 1. In the neural network video encoding unit 130, a trained neural network model is derived.

[0082] In step S14, the pre-processing unit 110 performs pre-processing on the input video using a neural network model (e.g., a first neural network model or a pre-processing model), and outputs the pre-processed video signal to the neural network video coding unit 130. The neural network video coding unit 130 performs coding on the pre-processed video signal using the trained neural network model, and outputs a neural network-coded video signal.

[0083] In step S15, the error determining unit 145 determines an error between the neural-network-encoded image and the input image, and optimizes parameters of the neural network model of the pre-processing unit 110. In the pre-processing unit 110, a trained neural network model is derived.

[0084] In step S16, the encoding device 100 ends the learning.

[0085] FIG. 8 is a diagram illustrating an example of an operation during inference according to the first embodiment.

[0086] As shown in FIG. 8, in step S20, the encoding device 100 starts inference.

[0087] In step S21, the pre-processing unit 110 performs pre-processing on the input video using a trained neural network model, and outputs a pre-processed video signal.

[0088] In step S22, the video encoding unit 120 performs encoding processing on the pre-processed video signal in accordance with a video encoding standard, and outputs an encoded video signal. The video encoding unit 120 may transmit the encoded video signal to the video decoding device as a bit stream.

[0089] In step S23, the encoding device 100 ends the inference.

[0090] [Second embodiment] Next, a second embodiment will be described.

[0091] In the first embodiment, an example has been described in which the encoding device 100 includes the pre-processing unit 110. In the second embodiment, an example will be described in which a pre-processing device having the pre-processing unit 110 is provided in the front stage of the encoding device 100.

[0092] FIG. 9 is a diagram illustrating an example of the configuration of an encoding system 10 according to the second embodiment.

[0093] 9, the encoding system 10 includes a pre-processing device 200 and an encoding device 100. The pre-processing device 200 includes a pre-processing unit 110. In addition, the encoding device 100 includes a video encoding unit 120.

[0094] The encoding system 10 shown in Fig. 9 shows an example of a configuration at the time of inference, similar to Fig. 1. That is, the pre-processing unit 110 performs pre-processing using an already trained neural network model (e.g., a first neural network model). Specifically, the pre-processing unit 110 performs pre-processing on a video signal of an input video (e.g., a first video signal) using the trained neural network model, and outputs the pre-processed video signal to the encoding device 100.

[0095] As in the first embodiment, the encoding device 100 performs encoding processing on the pre-processed video signal using a video encoding standard, and outputs a video signal of encoded video.

[0096] FIG. 10 is a diagram illustrating an example of the configuration of the encoding system 10 during learning according to the second embodiment.

[0097] As shown in FIG. 10, during learning, the pre-processing device 200 has a pre-processing unit 110, and the encoding device 100 has a neural network video encoding unit 130 and an error determining unit 145.

[0098] As in the first embodiment (FIG. 6), the pre-processing unit 110 performs pre-processing on the video signal of the input video using a neural network model (e.g., a first neural network model or a pre-processing model), and outputs the pre-processed video signal.

[0099] The neural network video encoding unit 130 has a trained neural network model (e.g., a second neural network model or a video encoding model) that has been trained using the neural network code amount and the encoding code amount, as in the first embodiment. As in the first embodiment, the neural network video encoding unit 130 performs encoding processing on the video signal of the pre-processed video using the neural network model, and outputs the video signal of the neural network encoded video.

[0100] Similarly to the first embodiment, the error determining unit 145 also optimizes the parameters of the neural network used in the pre-processing unit 110 using the loss function L2 shown in Equation 2.

[0101] In the second embodiment as well, the learning of the pre-processing model is performed using a trained video coding model. Moreover, the learning of the video coding model is performed based on the amount of encoded code and the amount of neural network code. Therefore, in the second embodiment as well, it is possible to perform learning so that not only the neural network encoded video approaches the encoded video, but also the ratio of the amount of encoded code and the amount of neural network code approaches a constant value. This makes it possible to, for example, make the accuracy of the video coding model derived by learning higher than a certain level. Moreover, since the pre-processing model is learned using the video coding model, it is possible to make the accuracy of the pre-processing model derived by learning higher than a certain level.

[0102] [Other embodiments] In the first and second embodiments, an example has been described in which the pre-processing unit 110 is trained after the neural network video encoding unit 130 is trained, but after the pre-processing unit 110 is trained, the neural network video encoding unit 130 may be trained again, and the pre-processing unit 110 may then be trained again. In other words, the neural network video encoding unit 130 may be trained and then the pre-processing unit 110 may be trained repeatedly.

[0103] A program may be provided that causes a computer to execute each process performed by the above-mentioned devices (encoding device 100 and pre-processing device 200). The program may be recorded in a computer-readable medium. The computer-readable medium can be used to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Furthermore, circuits that execute each process performed by the above-mentioned devices (encoding device 100 and pre-processing device 200) may be integrated, and the device may be configured as a semiconductor integrated circuit (chip set, SoC).

[0104] Although the embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes can be made without departing from the scope of the invention. In addition, each operation example can be combined without any contradiction. [Explanation of symbols]

[0105] 100: Encoding device 110: Pre-processing unit 120: Video encoding unit 121: Intra-coding unit 122: Inter-encoding unit 123: Encoding amount calculation unit 130: Neural network video coding unit 131: Neural network intra coding unit 132: Neural network inter-encoding unit 133: Neural network code amount calculation unit 140, 145: Error determination unit 200: Pre-processing device

Claims

1. a pre-processing unit that performs pre-processing on the first video signal by using the trained first neural network model and outputs the pre-processed video signal; a video encoding unit that performs an encoding process on the pre-processed video signal and outputs an encoded video signal, the training of the first neural network model is performed using a neural network video encoding unit including a trained second neural network model; The second neural network model is trained based on a code amount of an encoded video signal encoded by the video encoding unit for the second video signal and a code amount of a neural network-encoded video signal encoded by the neural network video encoding unit for the second video signal. Encoding device.

2. The neural network video encoding unit performs intra-encoding and inter-encoding on the second video signal using the second neural network model, and outputs the neural network-encoded video signal.

2. The encoding device according to claim 1.

3. a first error determination unit that optimizes parameters of the second neural network model by using a code amount of the encoded video signal and a code amount of the neural network encoded video signal when training the second neural network model.

3. The encoding device according to claim 2.

4. The first error determining unit optimizes the parameters of the second neural network model by using a loss function including the code amount of the encoded video signal and the code amount of the neural network encoded video signal.

4. The encoding device according to claim 3.

5. The loss function includes a scaling factor calculated from a ratio between the code amount of the encoded video signal and the code amount of the neural network encoded video signal.

5. The encoding device according to claim 4.

6. and a second error determination unit that performs error determination between a neural network-encoded video signal, which is an output of the neural network video encoding unit for the pre-processed video signal, and a third video signal input to the pre-processing unit, when the first neural network model is trained, and optimizes parameters of the first neural network model.

2. The encoding device according to claim 1.

7. An encoding program to be executed by a computer of an encoding device, performing pre-processing on the first video signal using the trained first neural network model, and outputting the pre-processed video signal; a process of encoding the pre-processed video signal and outputting the encoded video signal; the training of the first neural network model is performed using a neural network video encoding unit including a trained second neural network model; The second neural network model is trained based on a code amount of an encoded video signal encoded by the video encoding unit for the second video signal and a code amount of a neural network-encoded video signal encoded by the neural network video encoding unit for the second video signal. Encoding program.

8. A pre-processing device, a pre-processing unit that performs pre-processing on the first video signal by using the trained first neural network model and outputs the pre-processed video signal to the encoding device; The encoding device performs an encoding process on the pre-processed video signal, and outputs the encoded video signal; the training of the first neural network model is performed using a neural network video encoding unit including a trained second neural network model; The second neural network model is trained based on a code amount of an encoded video signal encoded by the video encoding unit for the second video signal and a code amount of a neural network-encoded video signal encoded by the neural network video encoding unit for the second video signal. Pre-processing device.

9. A pre-processing program executed by a computer of a pre-processing device, performing pre-processing on the first video signal using the trained first neural network model, and outputting the pre-processed video signal to an encoding device; The encoding device performs an encoding process on the pre-processed video signal, and outputs the encoded video signal; the training of the first neural network model is performed using a neural network video encoding unit including a trained second neural network model; The second neural network model is trained based on a code amount of an encoded video signal encoded by the video encoding unit for the second video signal and a code amount of a neural network-encoded video signal encoded by the neural network video encoding unit for the second video signal. Pre-processing program.

Images