Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

In order to ensure interconnectivity between a video encoding device and a video decoding device for executing a process using a neural network in the present invention, the video encoding device comprises: an encoding means that encodes a video signal; and a signaling means that signals information that can identify whether the process using the neural network and executed by the video decoding device is permitted.

Description

Video Encoding Device, Video Decoding Device, Video Encoding Method, and Video Decoding Method

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method.

[0002] In order to efficiently transmit or record video, a video encoding device that generates an encoded representation (hereinafter referred to as a bitstream) by encoding an input video, and a video decoding device that decodes the bitstream to generate a decoded video are used.

[0003] As standards for video encoding methods, there are H.265 / HEVC (High-Efficiency Video Coding) (see Non-Patent Document 1), H.266 / VVC (Versatile Video Coding) (see Non-Patent Document 2), etc., which are standardized by ITU-T SG16 and ISO / IEC / SC29.

[0004] In these video encoding methods, video data is managed in a hierarchical structure and encoded and decoded. The hierarchical structure consists of, for example, pictures that make up the video data, slices (or tiles) obtained by dividing a picture, coding tree units (CTUs) obtained by dividing a slice, and coding units (CUs) obtained by dividing a coding tree unit.

[0005] The input image of the processing target CU is usually predicted and encoded based on a predicted image that is encoded and decoded before the processing target CU and is generated based on a decoded image. That is, the prediction error image obtained by subtracting the predicted image from the input image is encoded and decoded. As prediction encoding, there are intra prediction (intra prediction) that uses a decoded image included in a picture at the same display time as the processing target CU, and inter prediction (inter prediction) that uses a decoded image included in a picture at a different display time from the processing target CU.

[0006] A CU encoded based on intra prediction is called an intra CU. A CU encoded based on inter prediction is called an inter CU.

[0007] A picture that can only contain an intraCU is called an I-picture. A picture that can contain not only an intraCU but also an interCU that uses a decoded image contained in one picture with a different display time than the target CU is called a P-picture. A picture that can contain not only an intraCU and an interCU that uses a decoded image contained in one picture with a different display time than the target CU, but also an interCU that uses decoded images contained in two pictures with different display times than the target CU is called a B-picture.

[0008] The coding of the prediction error image is performed based on frequency conversion, quantization, and entropy coding. The decoding of the prediction error image is performed based on entropy decoding, inverse quantization, and inverse frequency conversion.

[0009] Recommendation ITU-T H.264 "High efficiency video coding", Telecommunication Standardization Sector of ITU, April 2015 Recommendation ITU-T H.266 "Versatile video coding", Telecommunication Standardization Sector of ITU, August 2020

[0010] Furthermore, development of encoding schemes for machines used in machine recognition is underway. For example, ISO / IEC / SC29 is working on standardizing a video encoding scheme for machines called Video for Coding Machines (VCM). The following forms of use for the aforementioned video encoding are conceivable.

[0011] The video encoder applies preprocessing to the input video before encoding it using the video encoding method described above (hereinafter referred to as internal video encoding). The video encoder also encodes control information for the preprocessing. Preprocessing includes Temporal sampling, which reduces the number of pictures to encode by downsampling the pictures in the input video; Targeting, which reduces the region of interest (ROI) of a picture and grays out the rest of the picture; Spatial sampling, which reduces the size of the entire picture; and Bit depth truncation, which discards the lower bit information of the pixel values ​​of the picture. In bit depth truncation, for example, the lower bit information of the luminance component is discarded.

[0012] The video decoder decodes the bitstream using the video encoding method described above (hereinafter referred to as internal video decoding). The video decoder then applies post-processing to the decoded picture to obtain the final decoded image. The video decoder also decodes control information for post-processing. Post-processing includes temporal resampling, which interpolates the picture that was downsampled by time sampling in the video encoder; retargeting, which enlarges the gaze area that was reduced by targeting back to its original size; spatial resampling, which enlarges the picture that was reduced by time sampling back to its original size; and bit depth shift, which restores the brightness of the lower bits that were discarded by bit depth truncation.

[0013] Time resampling is the process corresponding to time sampling and is equivalent to the inverse process of time sampling. Retargeting is the process corresponding to targeting and is equivalent to the inverse process of targeting. Spatial sampling is the process corresponding to spatial sampling and is equivalent to the inverse process of spatial sampling. Bit depth shift is the process corresponding to bit depth truncation and is equivalent to the inverse process of bit depth truncation.

[0014] Video decoders incorporating neural networks are being considered. For example, a video decoder that uses a neural network to perform time-resampling is being envisioned.

[0015] Generally, neural networks are implemented using processors such as GPUs (Graphics Processing Units) and CPUs (Central Processing Units). Therefore, a video decoder incorporating a neural network will have a processor implemented to realize the neural network. When a neural network is used for time resampling, the processor generates an interpolated image from the input image (the image after decimation) using the neural network.

[0016] Sometimes, video decoders are equipped with processors that cannot perform neural network processing. In such cases, assuming that time resampling using a neural network is performed, the video decoder will not be able to reproduce the video when it receives a bitstream based on the video signal (video bitstream).

[0017] Furthermore, in video decoders equipped with a processor capable of neural network processing, the processor's performance may sometimes be insufficient for performing time-based resampling using a neural network.

[0018] For example, assuming time resampling is performed using a floating-point arithmetic unit, if the video bitstream is transmitted to a video decoder that does not have a floating-point arithmetic unit (for example, one that can only perform integer operations), the image quality of the video reproduced by the video decoder will deteriorate. In addition, the interpolated image generated by time resampling may deviate from the image input to the video encoding device.

[0019] The present invention aims to provide a video encoding device, a video decoding device, a video encoding method, a video decoding method, a video encoding program, and a video decoding program that can ensure interoperability between a video decoding device that performs processing using a neural network and a video encoding device.

[0020] The video encoding method based on this disclosure encodes a video signal and signals information that can identify whether or not processing using a neural network performed by a video decoding device is permitted.

[0021] Another aspect of the video encoding method based on this disclosure involves encoding a video signal and signaling information that can identify whether or not decimal values ​​may be rounded to integers in a process that utilizes a neural network performed by a video decoding device.

[0022] The video decoding method based on this disclosure decodes a video bitstream to obtain a video signal and determines whether or not to output the video signal based on signaled information that can identify whether or not processing using a neural network is permitted.

[0023] Another embodiment of the video decoding method based on this disclosure decodes a video bitstream to obtain a video signal and determines whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in a signaled neural network-based process.

[0024] The video encoding device based on this disclosure includes encoding means for encoding a video signal and signaling means for signaling information that can identify whether or not processing using a neural network performed by a video decoding device is permitted.

[0025] Other embodiments of the video encoding apparatus based on this disclosure include encoding means for encoding a video signal and signaling means for signaling information that can determine whether or not decimal values ​​may be rounded to integers in processing that utilizes a neural network performed by a video decoding apparatus.

[0026] The video decoding device according to this disclosure includes decoding means for decoding a video bitstream to obtain a video signal, and control means for determining whether or not to output a video signal based on signaled information that can identify whether or not processing using a neural network is permitted.

[0027] Other embodiments of the video decoding apparatus based on this disclosure include decoding means for decoding a video bitstream to obtain a video signal, and control means for determining whether or not to output a video signal based on signaled information that can identify whether or not decimal values ​​may be rounded to integers in a neural network-based process.

[0028] The video encoding program based on this disclosure causes a computer to perform the following tasks: encoding a video signal and signaling information that can determine whether or not processing using a neural network performed by a video decoding device is permitted.

[0029] Other embodiments of the video encoding program based on this disclosure cause a computer to perform the following: a process of encoding a video signal and a process of signaling information that can determine whether or not decimal values ​​may be rounded to integers in a process that utilizes a neural network performed by a video decoding device.

[0030] The video decoding program based on this disclosure performs the following processes: decoding a video bitstream to obtain a video signal, and determining whether or not to output the video signal based on signaled information that can identify whether or not processing using a neural network is permitted.

[0031] Other embodiments of the video decoding program based on this disclosure cause a computer to perform a process that signals information that can determine whether or not decimal values ​​may be rounded to integers in a process that utilizes a neural network performed by a video decoding device.

[0032] According to the present invention, interoperability between a video decoding device that performs processing using a neural network and a video encoding device is ensured.

[0033] This is a block diagram showing a video encoder and video decoder that encode and decode each picture that makes up video data. This is a block diagram showing an example of the configuration of a pre-processing unit and a post-processing unit. This is an explanatory diagram showing an example of time sampling processing. This is an explanatory diagram showing an example of time resampling processing. This is an explanatory diagram showing an example of an extended sequence parameter set. This is a flowchart showing an example of the operation of a video encoder. This is a flowchart showing an example of the operation of a video decoder. This is a block diagram showing an example of the configuration of an information processing system. This is a block diagram showing the main parts of a video encoding device. This is a block diagram showing the main parts of a video decoding device.

[0034] The embodiments will be described below with reference to the drawings.

[0035] Figure 1 is a block diagram showing a video encoder 100 and a video decoder 200 that encode and decode each picture that makes up the video data.

[0036] The video encoder 100 includes a pre-processing unit 101, an internal video encoder (internal encoder) 102, a multiplexer 103, and a control unit 104.

[0037] The preprocessing unit 101 performs preprocessing. The internal video encoder 102 encodes the video signal supplied from the preprocessing unit 101 and outputs a video bitstream. The control unit 104 generates an auxiliary information bitstream containing control information for post-processing in the video decoder 200. The multiplexer 103 multiplexes the video bitstream and the auxiliary information bitstream and outputs it. Therefore, the control information for post-processing is signaled to the video decoder 200.

[0038] The video decoder 200 includes a post-processing unit 201, an internal video decoder (internal decoder) 202, a demultiplexer 203, and a control unit 204.

[0039] The demultiplexer 203 demultiplexes the bit stream to obtain a video bit stream and an auxiliary information bit stream. The internal video decoder 202 decodes the video bit stream supplied from the demultiplexer 203 to obtain a decoded video (internal decoded video) of the internal video encoding. The control unit 204 decodes the auxiliary information bit stream supplied from the demultiplexer 203 to obtain control information for post-processing. The post-processing unit 201 executes post-processing.

[0040] Note that the arrows in FIG. 1 clearly indicate the direction of the signal (data) flow, but do not exclude bidirectionality. This also applies to other block diagrams.

[0041] [Explanation of Pre-Processing and Post-Processing] FIG. 2 is a block diagram showing a configuration example of the pre-processing unit and the post-processing unit.

[0042] In the example shown in FIG. 2, the pre-processing unit 101 includes a time sampling unit 1011, a targeting unit 1012, a spatial sampling unit 1013, and a bit depth reduction unit 1014. The output of the bit depth reduction unit 1014 is supplied to the internal video encoder 102. The auxiliary information is supplied to the multiplexer 103.

[0043] The time sampling unit 1011 performs the time sampling described above. The targeting unit 1012 performs the targeting process described above. The spatial sampling unit 1013 performs the spatial sampling described above. The bit depth reduction unit 1014 performs the bit depth reduction described above.

[0044] The post-processing unit 201 includes a spatial resampling unit 2011, a re-targeting unit 2012, a time resampling unit 2013, and a bit depth shift unit 2014.

[0045] The spatial resampling unit 2011 performs the spatial resampling described above. The re-targeting unit 2012 performs the re-targeting process described above. The time resampling unit 2013 performs the time resampling described above. The bit depth shift unit 2014 performs the bit depth shift described above.

[0046] Next, time sampling and time resampling will be described in more detail.

[0047] Time sampling thins out the pictures of the input video to reduce the number of pictures to be encoded by the internal video encoding. As a result, the effect of reducing the amount of codes in the bit stream can be obtained. Time resampling is a process of interpolating the pictures thinned out by time sampling in multiple stages. By this process, the temporal resolution for the thinned-out pictures is restored.

[0048] Note that the maximum number of skipped pictures MaxTPics by time sampling is defined by, for example, the following formula T1.

[0049] MaxTPics = (2^(max_temporal_resampling_ratio_idx_minus1+1)) - 1 (Formula T1)

[0050] max_temporal_resampling_ratio_idx_minus1 is a parameter (syntax element) multiplexed into the bit stream as will be described later.

[0051] As can be seen from Formula T1, max_temporal_resampling_ratio_idx_minus1 can be expressed by the following formula T2.

[0052] max_temporal_resampling_ratio_idx_minus1 = log2(MaxTPics + 1) (Formula T2)

[0053] FIG. 3 is an explanatory diagram showing an example of the time sampling process executed by the time sampling unit 1011. FIG. 4 is an explanatory diagram showing an example of the time resampling process executed by the time resampling unit 2013. Note that FIGS. 3 and 4 illustrate the case where max_temporal_resampling_ratio_idx_minus1 = 1, that is, the case where MaxTPics = 3.

[0054] As illustrated in Figure 3, the amount of encoded data is reduced by performing internal encoding after the picture has been downsampled by time sampling. Furthermore, as illustrated in Figure 4, the picture downsampled in the video encoder 100 is restored by time resampling, for example, multi-stage interpolation.

[0055] Parameters related to time sampling and time resampling can be multiplexed into the bitstream using the extended sequence parameter set illustrated in Figure 5.

[0056] In the example shown in Figure 5, temporal_resampling_flag=1 indicates that temporal sampling / resampling is enabled. temporal_resampling_flag=0 indicates that temporal sampling / resampling is disabled.

[0057] max_temporal_resampling_ratio_idx_minus1 is used to derive the variable TemporalResamplingRatio as follows: In equation T3, the value of max_temporal_resampling_ratio_idx_minus1 is in the range of 0 to 3.

[0058] TemporalResamplingRatio = 2(max_temporal_resampling_ratio_idx_minus1 + 1) (Formula T3)

[0059] The time sampling unit 1011 supplies the above-mentioned temporal_resampling_flag and max_temporal_resampling_ratio_idx_minus1 to the control unit 104 as auxiliary information.

[0060] [Explanation of the Encoding Side] In this embodiment, as described above, the preprocessing unit 101 supplies the internal video encoder 102 with a video signal that has undergone time sampling, targeting processing, spatial sampling, and bit depth truncation on the input video. The preprocessing unit 101 also supplies the parameters for the time sampling, targeting processing, spatial sampling, and bit depth truncation described above to the control unit 104 as auxiliary information.

[0061] Figure 6 is a flowchart showing an example of the operation of the video encoder 100. The operation of the video encoder 100 will be explained with reference to Figure 6.

[0062] In the video encoder 100, the preprocessing unit 101 performs the above-described preprocessing, namely time sampling, targeting, spatial sampling, and bit depth truncation (step S101).

[0063] In this embodiment, as described above, a neural network is utilized in the time resampling of the video decoder 200.

[0064] However, among the video decoding devices that receive the video bitstream from the video encoder 100, there is a possibility that some video decoding devices are equipped with processors that cannot perform neural network processing (in this embodiment, time resampling using a neural network), as described above. Furthermore, among video decoding devices equipped with processors capable of performing neural network processing, there is a possibility that some are equipped with processors that are not sufficiently powerful for executing neural network processing.

[0065] An example of a processor with insufficient performance for performing neural network processing is one that only has arithmetic units capable of performing integer operations. Another example of an insufficient processor is one that has floating-point arithmetic units, but whose number is significantly less than the number of arithmetic units capable of performing integer operations only.

[0066] Whether the number of floating-point arithmetic units is significantly less than the number of arithmetic units capable of performing integer arithmetic alone is determined, for example, by whether the quality of the video reproduced by the video decoder exceeds a predetermined quality.

[0067] Therefore, in the video encoder 100, the control unit 104 includes information (flags) indicating whether processing using a neural network is enabled or disabled in the auxiliary information, either in place of or in addition to the temporal_resampling_flag which indicates that the above-mentioned temporal sampling / temporal resampling is enabled.

[0068] For example, the control unit 104 enables information indicating processing that utilizes a neural network if all video decoding devices that receive the video bitstream from the video encoder 100 are equipped with processors that have sufficient performance to execute neural network processing. If there are video encoding devices that are equipped with processors that have insufficient performance to execute neural network processing, the control unit 104 disables information indicating processing that utilizes a neural network.

[0069] For example, in a video decoding device equipped with a processor that only has arithmetic units capable of performing integer operations, it is conceivable that decimal values ​​would be rounded to integers before performing integer operations. The quality of the video reproduced through such operations would naturally deteriorate.

[0070] However, if a certain degree of degradation in video quality is acceptable, the video encoder 100 may allow rounding to integers in the calculations during neural network processing. For example, auxiliary information indicating that rounding to integers is permitted in the calculations during neural network processing is defined. The video encoder 100 then enables this auxiliary information when the video decoder to which the video bitstream is transmitted includes a video decoder equipped with a processor that has only an arithmetic processing unit capable of performing integer calculations, and when rounding decimal values ​​to integers in the calculations during neural network processing results in a degradation of video quality below an acceptable level.

[0071] The internal video encoder 102 encodes the bit-depth shifted video data supplied from the preprocessing unit 101 to generate a video bitstream (step S102).

[0072] The control unit 104 generates an auxiliary information bitstream (step S103). The auxiliary information bitstream includes, as auxiliary information, the parameters of each process in the preprocessing and parameters that can identify the processing order of each process constituting the postprocessing. The multiplexer 103 multiplexes the video bitstream and the auxiliary information bitstream (step S104).

[0073] [Decoding Explanation] Figure 7 is a flowchart showing an example of the operation of the video decoder 200. The operation of the video decoder 200 will be explained with reference to Figure 7.

[0074] In the video decoder 200, the demultiplexer 203 demultiplexes the bitstream, supplies the video bitstream to the internal video decoder 202, and supplies the auxiliary information bitstream to the control unit 204 (step S201). The internal video decoder 202 decodes the video bitstream to obtain the decoded video (step S202).

[0075] The control unit 204 obtains auxiliary information related to post-processing from the auxiliary information bitstream (step S203). Each block constituting the post-processing unit 201 (spatial resampling unit 2011, retargeting unit 2012, time resampling unit 2013, and bit depth shift unit 2014) performs post-processing using the signaled auxiliary information related to post-processing (step S204).

[0076] The control unit 204 determines that if the information in the auxiliary information bitstream indicating whether processing using a neural network is enabled or disabled indicates disabled, time resampling is not permitted and video playback is not possible. When this determination is made, the post-processing unit 201 does not output the decoded video. Furthermore, when this determination is made, the control unit 204 does not have the post-processing unit 201 perform post-processing. Moreover, when this determination is made, the internal video decoder 202 does not perform playback of the video signal by decoding processing. In other words, the control unit 204 determines whether or not to output the video signal based on the information indicating whether processing using a neural network is enabled or disabled.

[0077] Furthermore, if the video decoder 200 has a logic unit capable of performing only integer operations with respect to neural network processing, the following control is performed.

[0078] In other words, if the signaled auxiliary information indicating that rounding to an integer is permitted in the calculations in the neural network processing is valid, the control unit 204 permits the execution of time resampling by the time resampling unit 2013. The time resampling unit 2013 performs the neural network processing after rounding the decimal part of the value to an integer in the calculations, if the processor built into the video decoder 200 is a processor that only has an arithmetic processing unit capable of performing integer calculations.

[0079] If the auxiliary information indicating that rounding to an integer is permitted in the calculations in the neural network processing is invalid, the control unit 204 does not need to have the post-processing unit 201 perform post-processing. Furthermore, if it is determined that this is the case, the internal video decoder 202 may not perform playback of the video signal by decoding. In other words, the control unit 204 determines whether or not to output the video signal based on the auxiliary information indicating that rounding to an integer is permitted in the calculations in the neural network processing.

[0080] As described above, in this embodiment, the interoperability between the video decoder and the video encoder that perform processing using a neural network in the VCM can be ensured. For example, situations in which the video decoder outputs low-quality decoded video or becomes unable to restore the decoded video can be avoided. Therefore, in a video system in which a video encoder and multiple video decoders exist, the video encoder can set an appropriate operation point. Setting an appropriate operation point means that each of the multiple video decoders does not experience any problems with the decoding process or output an inappropriate decoded video. In other words, setting an appropriate operation point means that processing is performed in the most desirable state in the video system.

[0081] In the above embodiment, time resampling was used as an example of a process that utilizes a neural network. However, if there are other processes that utilize a neural network in the video decoding device, the concept of the above embodiment can be applied to those processes as well.

[0082] Furthermore, in the above embodiment, the internal decoded video from the internal video encoder 102 and the output video from the internal video decoder 202 match, rather than the post-processing result of the internally decoded video from the internal video encoder 102 matching the final output video. In other words, the above embodiment can be interpreted as an embodiment that makes them match.

[0083] While the above embodiment can be implemented using hardware, it can also be realized using a computer program.

[0084] The information processing system shown in Figure 8 includes a processor 701 such as a CPU or GPU, a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. Multiple processors 701 may be provided. The storage mediums 703 and 704 may be separate storage mediums or may be storage areas consisting of the same storage medium. Magnetic storage media such as hard disks can be used as storage media. Furthermore, the information processing system may include multiple processors 701.

[0085] In the information processing system, the program memory 702 stores programs (video encoding programs or video decoding programs) for realizing the functions of each block shown in the above embodiment.

[0086] The processor 701 then performs processing according to the program stored in the program memory 702, thereby realizing the functions of the video encoder 100 or video decoder 200 shown in the above embodiment.

[0087] For example, the functions of the video encoder 100 are realized when the processor 701 executes processing according to a video encoding program that realizes the functions of each block in the video encoder 100 shown in Figure 1.

[0088] Furthermore, for example, the processor 701 executes processing according to a video decoding program that realizes the functions of each block in the video decoder 200 shown in Figure 1, thereby realizing the functions of the video decoder.

[0089] Furthermore, at least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transient computer-readable medium. The program is supplied to the transient computer-readable medium, for example, via a wired communication channel or a wireless communication channel, i.e., via electrical signals, optical signals or electromagnetic waves.

[0090] Figure 8 is a block diagram showing the main parts of a video encoding device. The video encoding device 10 shown in Figure 8 (implemented by a video encoder 100 in this embodiment) comprises encoding means 11 for encoding video signals (implemented by an internal video encoder 102 in this embodiment) and signaling means 12 for signaling information that can identify whether or not processing using a neural network executed by a video decoding device is permitted (implemented by a multiplexer 103 and a control unit 104 in this embodiment).

[0091] Figure 9 is a block diagram showing the main parts of the video decoding device. The video decoding device 20 shown in Figure 9 (implemented by a video decoder 200 in this embodiment) comprises decoding means 21 (implemented by an internal video decoder 202 in this embodiment) that decodes a video bitstream to acquire a video signal, and control means 22 (implemented by a control unit 204 in this embodiment) that determines whether or not to output a video signal based on signaled information that can identify whether or not processing using a neural network is permitted.

[0092] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0093] (Note 1) A video encoding method that signals information that can identify whether or not processing using a neural network executed by a video decoding device is permitted after encoding the video signal.

[0094] (Note 2) A video encoding method that signals information that can determine whether or not decimal values ​​should be rounded to integers in a process that uses a neural network to encode a video signal and is performed by a video decoding device.

[0095] (Note 3) The video encoding method described in Note 1, which signals information that can determine whether or not decimal values ​​may be rounded to integers in processing that utilizes a neural network performed by a video decoding device.

[0096] (Note 4) A video encoding method in which a predetermined preprocessing is performed before encoding the video signal, and the processing using the neural network is a processing included in the preprocessing that corresponds to time resampling which reduces the number of pictures, as specified in Note 1 to Note 3.

[0097] (Note 5) A video decoding method that decodes a video bitstream to obtain a video signal and determines whether or not to output the video signal based on signaled information that can identify whether or not processing using a neural network is permitted.

[0098] (Note 6) A video decoding method that decodes a video bitstream to obtain a video signal and determines whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in a signaled neural network-based processing.

[0099] (Note 7) The video decoding method described in Note 5, which determines whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in the signaled neural network-based processing.

[0100] (Note 8) A video decoding method in which a predetermined post-processing is applied to the decoded video signal, and the processing utilizing the neural network is a process included in the post-processing that interpolates the picture using time resampling, as specified in any of Notes 5 to 7.

[0101] (Note 9) A video encoding device comprising encoding means for encoding a video signal and signaling means for signaling information that can identify whether or not processing using a neural network executed by a video decoding device is permitted.

[0102] (Note 10) A video encoding device comprising encoding means for encoding a video signal, and signaling means for signaling information that can determine whether or not decimal values ​​may be rounded to integers in processing that utilizes a neural network performed by a video decoding device.

[0103] (Note 11) The video encoding device according to Note 9, wherein the signaling means signals information that can determine whether or not decimal values ​​may be rounded to integers in processing that utilizes a neural network performed by the video decoding device.

[0104] (Note 12) The video encoding device described in Note 9, wherein the signaling means signals information that can determine whether or not decimal values ​​may be rounded to integers in processing that utilizes a neural network performed by the video decoding device.

[0105] (Note 13) A video decoding device comprising: decoding means for decoding a video bitstream to obtain a video signal; and control means for determining whether or not to output the video signal based on signaled information that can specify whether or not decimal values ​​may be rounded to integers in processing that utilizes a neural network.

[0106] (Note 14) The control means is a video decoding device, as specified in Note 12, which determines whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in a signaled neural network-based processing.

[0107] (Note 15) A video encoding program that causes a computer to perform the process of encoding a video signal and the process of signaling information that can determine whether or not the process using a neural network executed by a video decoding device is permitted.

[0108] (Note 16) A video encoding program that causes a computer to perform the following processes: encoding a video signal and signaling information that can determine whether or not decimal values ​​should be rounded to integers in the neural network processing performed by the video decoding device.

[0109] (Note 17) The video encoding program described in Note 15, which causes a computer to perform a process that signals information that can determine whether or not decimal values ​​may be rounded to integers in a process that utilizes a neural network executed by a video decoding device.

[0110] (Note 18) A video decoding program that causes a computer to perform the following processes: decoding a video bitstream to obtain a video signal, and determining whether or not to output the video signal based on signaled information that can identify whether or not processing using a neural network is permitted.

[0111] (Note 19) A video decoding program that causes a computer to perform the following processes: decoding a video bitstream to obtain a video signal, and determining whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in a signaled neural network-based process.

[0112] (Note 20) The video decoding program described in Note 18, which causes a computer to perform a process to determine whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in a signaled neural network-based process.

[0113] The configuration described in Appendix 4, which is subordinate to Appendix 1 to Appendix 3 above, and some or all of Appendix 8, which is subordinate to Appendix 5 to Appendix 7, can be made subordinate to Appendix 9 to Appendix 11, Appendix 12 to Appendix 14, Appendix 15 to Appendix 17, and Appendix 18 to Appendix 20 in the same subordinate relationships as Appendix 4 and Appendix 8. Furthermore, not limited to Appendix 9 to Appendix 11, Appendix 12 to Appendix 14, Appendix 15 to Appendix 17, and Appendix 18 to Appendix 20, some or all of the configurations described as appendices above can be made subordinate to various hardware, software, various recording means for recording software, or systems, provided that they do not deviate from the embodiments described above.

[0114] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

[0115] This application claims priority based on Japanese Patent Application No. 2024-179877, filed on 15 October 2024, and incorporates all of its disclosures herein.

[0116] 10 Video encoding device 11 Encoding means 12 Signaling means 20 Video decoding device 21 Decoding means 22 Control means 100 Video encoder 101 Pre-processing unit 102 Internal video encoder 103 Multiplexer 104 Control unit 200 Video decoder 201 Post-processing unit 202 Internal video decoder 203 Demultiplexer 204 Control unit 701 Processor 702 Program memory 703, 704 Storage medium 1011 Time sampling unit 1012 Targeting unit 1013 Spatial sampling unit 1014 Bit depth truncation unit 2011 Spatial resampling unit 2012 Retargeting unit 2013 Time resampling unit 2014 Bit depth shift unit

Claims

1. A video encoding method that signals information that can identify whether or not processing using a neural network executed by a video decoding device is permitted after encoding the video signal.

2. A video encoding method that signals information that can determine whether or not decimal values ​​should be rounded to integers in a process that uses a neural network to encode a video signal and is performed by a video decoding device.

3. The video encoding method according to claim 1, which signals information that can determine whether or not decimal values ​​may be rounded to integers in a process that utilizes a neural network performed by a video decoding device.

4. A video encoding method according to any one of claims 1 to 3, wherein a predetermined preprocessing is performed before encoding the video signal, and the processing using the neural network is a process included in the preprocessing that corresponds to time resampling for time sampling to reduce the number of pictures.

5. A video decoding method that decodes a video bitstream to obtain a video signal and determines whether or not to output the video signal based on signaled information that can identify whether or not processing using a neural network is permitted.

6. A video decoding method that decodes a video bitstream to obtain a video signal, and determines whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in a signaled neural network-based processing.

7. The video decoding method according to claim 5, which determines whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in a signaled neural network-based processing.

8. A video decoding method according to any one of claims 5 to 7, wherein a predetermined post-processing is applied to the decoded video signal, and the processing utilizing the neural network is a process included in the post-processing that interpolates the picture using time resampling.

9. A video encoding device comprising encoding means for encoding a video signal, and signaling means for signaling information that can identify whether or not processing using a neural network executed by a video decoding device is permitted.

10. A video encoding device comprising encoding means for encoding a video signal, and signaling means for signaling information that can determine whether or not decimal values ​​may be rounded to integers in processing that utilizes a neural network performed by a video decoding device.

11. The video encoding device according to claim 9, wherein the signaling means signals information that can determine whether or not decimal values ​​may be rounded to integers in processing that utilizes a neural network performed by the video decoding device.

12. A video decoding device comprising: decoding means for decoding a video bitstream to obtain a video signal; and control means for determining whether or not to output the video signal based on signaled information that can identify whether or not processing using a neural network is permitted.

13. A video decoding device comprising: decoding means for decoding a video bitstream to obtain a video signal; and control means for determining whether or not to output the video signal based on signaled information that can identify whether or not decimal values ​​may be rounded to integers in processing that utilizes a neural network.

14. The video decoding device according to claim 12, wherein the control means determines whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in a signaled neural network-based processing.

15. A video encoding program that causes a computer to perform the processes of encoding a video signal and signaling information that can determine whether or not the processing using a neural network performed by a video decoding device is permitted.

16. A video encoding program that causes a computer to perform the following processes: encoding a video signal and signaling information that can determine whether or not decimal values ​​should be rounded to integers in the neural network processing performed by the video decoding device.

17. The video encoding program according to claim 15, which causes a computer to perform a process of signaling information that can determine whether or not decimal values ​​may be rounded to integers in a process that utilizes a neural network performed by a video decoding device.

18. A video decoding program that causes a computer to perform the following processes: decoding a video bitstream to obtain a video signal, and determining whether or not to output the video signal based on signaled information that can identify whether or not processing using a neural network is permitted.

19. A video decoding program that causes a computer to perform the following processes: decoding a video bitstream to obtain a video signal; and determining whether or not to output the video signal based on signaled information that can identify whether or not decimal values ​​may be rounded to integers in a neural network-based process.

20. The video decoding program according to claim 18, which causes a computer to perform a process to determine whether or not to output the video signal based on information that can identify whether or not decimal values ​​may be rounded to integers in a signaled neural network-based process.

Images