Encoding device and encoding method

Abstract

An encoding device (100) comprises a processor (140) and an encoder (120) that encodes a frame to be encoded in units of blocks. The processor (140) acquires an index of visibility of an image quality difference between blocks in the frame to be encoded, determines whether to apply intra refresh to the frame to be encoded on the basis of the acquired index, and controls the encoder (120) to encode the frame to be encoded on the basis of the result of the determination.

Description

Encoding device and encoding method

[0001] The present disclosure relates to an encoding device and an encoding method.

[0002] Common video coding standards (e.g., H.264 (MPEG-4 AVC), H.265 (HEVC), H.266 (VVC), VP9, ​​AV1, etc.) employ techniques for compressing data by exploiting data redundancy between frames. As a result, data loss in a frame causes degradation of image quality in other frames that directly or indirectly reference that frame. This chain of image quality degradation between frames is eliminated by using frames that are intra-coded without using a reference frame (hereinafter referred to as intra-frames). However, with intra-frames, the increased data volume increases transmission delays.

[0003] Therefore, in order to limit the chain of image quality degradation with lower delay than intra-frame, intra-refresh has been proposed, which forcibly intra-codes a part of a frame (see, for example, Patent Document 1).

[0004] Special Publication No. 2008-516561

[0005] However, in the case of intra-refresh, there are cases where the image quality is significantly degraded due to some areas that are forcibly intra-coded (hereinafter referred to as intra areas).

[0006] Therefore, the present disclosure provides an encoding device and the like that can suppress image quality degradation due to intra-refresh.

[0007] An encoding device according to one aspect of the present disclosure includes a processor and an encoder that encodes a frame to be encoded on a block-by-block basis, wherein the processor acquires an index of visibility of image quality differences between blocks within the frame to be encoded, determines whether to apply intra-refresh to the frame to be encoded based on the acquired index, and controls the encoder to encode the frame to be encoded based on the result of the determination.

[0008] This comprehensive or specific aspect may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium, or as any combination of a system, a device, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory).

[0009] According to the present disclosure, it is possible to suppress image quality degradation due to intra-refresh.

[0010] FIG. 1 is an external view of a coding / decoding system according to an embodiment. FIG. 2 is a functional configuration diagram of the coding / decoding system according to an embodiment. FIG. 3 is a diagram showing an example of a video sequence for explaining intra refresh. FIG. 4 is a flowchart showing a coding method according to an embodiment. FIG. 5 is a flowchart showing intra frame determination processing according to an embodiment. FIG. 6 is a flowchart showing intra refresh determination processing according to an embodiment. FIG. 7 is a flowchart showing frame coding processing according to an embodiment. FIG. 8 is a diagram showing an example of a video sequence coded by a coding device according to an embodiment.

[0011] (Summary of the Present Disclosure) Before describing the embodiments, an overview of the present disclosure will be described. Intra refresh compulsorily intra-codes a portion of a frame. Therefore, image quality degradation becomes noticeable when block distortion between an intra-coded block and a block coded using another method is easily visible. In other words, image quality degradation due to intra refresh becomes noticeable when image quality differences between blocks within a frame are easily visible.

[0012] An encoding device according to a first aspect of the present disclosure includes a processor and an encoder that encodes a frame to be encoded on a block-by-block basis, wherein the processor acquires an index of visibility of image quality differences between blocks in the frame to be encoded, determines whether to apply intra-refresh to the frame to be encoded based on the acquired index, and controls the encoder to encode the frame to be encoded based on the result of the determination.

[0013] According to this, whether to apply intra-refresh to a frame to be coded is determined based on an index of visibility of differences in image quality between blocks in the frame to be coded. Therefore, it is possible to switch between applying and not applying intra-refresh on a frame-by-frame basis depending on the visibility of differences in image quality between blocks. As a result, it is possible to suppress image quality degradation due to differences in image quality between a region that is forcibly intra-coded by intra-refresh and other regions.

[0014] A coding device according to a second aspect of the present disclosure is a coding device according to the first aspect, wherein the processor obtains the value of a quantization parameter of the frame to be coded as the index, and determines to apply the intra-refresh to the frame to be coded if the value of the quantization parameter is less than a first threshold, and determines not to apply the intra-refresh to the frame to be coded if the value of the quantization parameter is greater than or equal to the first threshold.

[0015] According to this, the value of the quantization parameter can be used as an index of the visibility of the difference in image quality between blocks. Generally, the quantization parameter is used for bitrate control. Specifically, by increasing the value of the quantization parameter, the size of the quantization step when quantizing a prediction error signal can be increased, thereby reducing the bitrate. However, as the size of the quantization step increases, the amount of information lost from the prediction error signal due to quantization (quantization distortion) increases, thereby increasing the visibility of the difference in image quality between an intra-coded block and an inter-coded block. Therefore, by determining not to apply intra-refresh to the frame to be coded when the value of the quantization parameter is equal to or greater than a first threshold, it is possible to not apply intra-refresh when the visibility of the difference in image quality between blocks is high. As a result, it is possible to prevent the visibility of the difference in image quality between a region that is forcibly intra-coded by intra-refresh and other regions from becoming high, thereby suppressing image quality degradation due to intra-refresh.

[0016] A coding device according to a third aspect of the present disclosure is a coding device according to the first aspect, wherein the processor acquires the value of the skip mode selection rate in the frame preceding the frame to be coded as the index, and determines not to apply the intra-refresh to the frame to be coded if the value of the skip mode selection rate is greater than or equal to a second threshold, and determines to apply the intra-refresh to the frame to be coded if the value of the skip mode selection rate is less than the second threshold.

[0017] According to this, the value of the skip mode selection rate can be used as an index of the visibility of the difference in image quality between blocks. Generally, in skip mode, the coding of a prediction error signal is omitted, which can reduce the bit rate but is likely to increase distortion. Therefore, when a low bit rate is required, the value of the skip mode selection rate increases, and the visibility of the difference in image quality between an intra-coded block and other blocks (including blocks coded in skip mode) becomes higher. Therefore, by determining not to apply intra-refresh to the coding target frame when the value of the skip mode selection rate is equal to or greater than a second threshold, it is possible to not apply intra-refresh when the visibility of the difference in image quality between blocks is high. As a result, it is possible to prevent the visibility of the difference in image quality between an area that is forcibly intra-coded by intra-refresh and other areas from becoming high, and to suppress image quality degradation due to intra-refresh.

[0018] A coding device according to a fourth aspect of the present disclosure is a coding device according to the first aspect, wherein the processor acquires the value of the intra-mode selection rate in the frame preceding the frame to be coded as the index, and determines to apply the intra-refresh to the frame to be coded if the value of the intra-mode selection rate is greater than or equal to a third threshold, and determines not to apply the intra-refresh to the frame to be coded if the value of the intra-mode selection rate is less than the third threshold.

[0019] According to this, the value of the intra mode selection rate can be used as an index of the visibility of the difference in image quality between blocks. As the value of the intra mode selection rate increases, the number of blocks that are intra-coded increases, and the visibility of the difference in image quality between blocks that are forcibly intra-coded by intra refresh and other blocks decreases. Conversely, as the value of the intra mode selection rate decreases, the visibility of the difference in image quality between blocks that are forcibly intra-coded by intra refresh and other blocks increases. Therefore, by determining not to apply intra refresh to the encoding target frame when the value of the intra mode selection rate is less than the third threshold, it is possible to not apply intra refresh when the visibility of the difference in image quality between blocks is high. As a result, it is possible to prevent the visibility of the difference in image quality between areas that are forcibly intra-coded by intra refresh and other areas from becoming too high, and to suppress image quality degradation due to intra refresh.

[0020] A coding device according to a fifth aspect of the present disclosure is a coding device according to the first aspect, wherein the processor acquires the value of the amount of change between the frame to be coded and the frame preceding the frame to be coded as the index, and determines to apply the intra-refresh to the frame to be coded if the value of the amount of change is greater than or equal to a fourth threshold, and determines not to apply the intra-refresh to the frame to be coded if the value of the amount of change is less than the fourth threshold.

[0021] According to this, the value of the amount of change between frames can be used as an index of the visibility of the difference in image quality between blocks. When the value of the amount of change between frames is large, the prediction accuracy of inter prediction decreases, the difference between the prediction error of intra prediction and the prediction error of inter prediction decreases, and the visibility of the difference in image quality between blocks due to intra refresh decreases. Conversely, when the value of the amount of change between frames is small, the prediction accuracy of inter prediction increases, the difference between the prediction error of intra prediction and the prediction error of inter prediction increases, and the visibility of the difference in image quality between blocks due to intra refresh increases. Therefore, by determining not to apply intra refresh to the frame to be coded when the value of the amount of change is less than the fourth threshold, it is possible to not apply intra refresh when the visibility of the difference in image quality between blocks is high. As a result, it is possible to prevent the visibility of the difference in image quality between an area that is forcibly intra-coded by intra refresh and other areas from becoming high, and to suppress image quality degradation due to intra refresh.

[0022] A coding device according to a sixth aspect of the present disclosure is a coding device according to any one of the first to fifth aspects, wherein the processor determines whether to apply intra-refresh in units of cycles for intra-refresh including the frame to be coded.

[0023] According to this, whether or not to apply intra-refresh is determined on a periodic basis, so that the load of the intra-refresh determination process can be reduced.

[0024] A coding device according to a seventh aspect of the present disclosure is the coding device according to the sixth aspect, wherein the current frame to be coded is a first frame in a cycle for the intra refresh.

[0025] This makes it possible to determine whether or not to apply intra-refresh at the first frame of the cycle for intra-refresh, thereby simplifying the intra-refresh determination process.

[0026] An encoding device according to an eighth aspect of the present disclosure is an encoding device according to any one of the first to seventh aspects, further comprising a communication interface for transmitting a plurality of encoding frames including the encoded encoding target frame via packet communication, and the processor further determines whether a recovery wait time is longer than a predetermined time when a recovery wait time occurs in the encoding target frame due to packet loss in communication of a frame prior to the encoding target frame, and sets the encoding target frame to an intra frame in response to determining that the waiting time is longer than the predetermined time.

[0027] According to this, an intra-frame is inserted when the recovery wait time is long, thereby preventing image quality degradation caused by packet loss from continuing for a long period of time.

[0028] An encoding method according to a ninth aspect of the present disclosure is an encoding method for encoding a frame to be encoded on a block-by-block basis, which obtains an index of visibility of image quality differences between blocks within the frame to be encoded, determines whether to apply intra-refresh to the frame to be encoded based on the obtained index, and encodes the frame to be encoded based on the result of the determination.

[0029] This makes it possible to realize the same effect as the encoding device described above with the encoding method.

[0030] A program according to a tenth aspect of the present disclosure causes a computer to execute the encoding method according to the ninth aspect.

[0031] This makes it possible to achieve the same effect as the encoding device described above by using a program.

[0032] (Embodiments) Hereinafter, embodiments will be specifically described with reference to the drawings.

[0033] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure.

[0034] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and duplicated explanations may be omitted or simplified.

[0035] Hereinafter, all or part of a circuit, unit (part), or device, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI or IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). Field programmable gate arrays (FPGAs), which are programmed after the LSI is manufactured, or reconfigurable logic devices (RLDs), which can reconfigure the connections within the LSI or set up circuit partitions within the LSI, can also be used for the same purpose.

[0036] Furthermore, all or part of a circuit, unit, or device, or all or part of a functional block in a block diagram, may be implemented by a software program. In this case, the software program is recorded on one or more non-transitory storage media such as a ROM, optical disk, hard disk drive, etc., and when the software program is executed by a processor, the functions specified in the software program are performed by the processor and peripheral devices. A system or device may include one or more non-transitory storage media in which a software program is stored, a processor, and necessary hardware devices (e.g., memory, interface, etc.).

[0037] [Configuration of Coding / Decoding System 1000] First, the configuration of the coding / decoding system 1000 according to this embodiment will be specifically described with reference to Fig. 1 and Fig. 2. Fig. 1 is an external view of the coding / decoding system 1000 according to this embodiment. Fig. 2 is a functional configuration diagram of the coding / decoding system 1000 according to this embodiment. Note that Figs. 1 and 2 show an exemplary configuration of the coding / decoding system 1000, and the configuration of the coding / decoding system 1000 is not limited to the following description.

[0038] The encoding / decoding system 1000 includes an encoding device 100 and a decoding device 200 that are wirelessly connected to each other. Note that the connection between the encoding device 100 and the decoding device 200 is not limited to a wireless connection. For example, the encoding device 100 may be connected to the decoding device 200 by wire in addition to or instead of a wireless connection.

[0039] The encoding device 100 is, for example, an HDMI (registered trademark) (High-Definition Multimedia Interface) sink device, and can encode video data received from an HDMI source device and transmit the encoded video data to the decoding device 200. Note that the encoding device 100 is not limited to an HDMI sink device. For example, the encoding device 100 may be a set-top box, a TV tuner, or a digital video camera.

[0040] The decoding device 200 is, for example, a liquid crystal display or an organic EL display, and can decode and display the encoded video data received from the encoding device 100. Note that the decoding device 200 may also be a portable device (e.g., a tablet computer or a smartphone) or a wearable device (e.g., a smart watch or smart glasses). In this case, the decoded video data may be displayed on a display panel of the portable device or the wearable device.

[0041] [Internal Configuration of Encoding Apparatus 100] Next, a description will be given of the internal configuration of the encoding apparatus 100. The encoding apparatus 100 includes an HDMI interface 110, an encoder 120, a communication interface 130, a processor 140, and a memory 150.

[0042] The HDMI interface 110 is an interface for receiving video data. The video data includes multiple frames. For example, the HDMI interface 110 can receive HDMI video data from an HDMI source device via an HDMI cable. Note that the HDMI interface 110 does not necessarily have to be included in the encoding device 100.

[0043] The encoder 120 can encode the video data received by the HDMI interface 110. Specifically, the encoder 120 can encode a frame to be encoded included in the video data in units of blocks. The encoder 120 can use intra-refresh when encoding the frame to be encoded. Intra-refresh is a technique that can limit a chain of image quality degradation with low latency by forcibly intra-coding a portion of a frame (an intra-area) and periodically changing the position of the intra-area across multiple frames.

[0044] A block is an area obtained by dividing a frame to be coded, and may also be called a macroblock, a coding unit (CU), or a prediction unit (PU) depending on the video coding standard.

[0045] The communication interface 130 is an interface for wireless communication with the decoding device 200. For example, the communication interface 130 can packetize the video data encoded by the encoder 120 and transmit the resulting communication data to the decoding device 200. Note that the communication interface 130 may be an interface for wired communication with the decoding device 200, and may not be included in the encoding device 100.

[0046] The processor 140 is a circuit that performs information processing and is a circuit that can access the memory 150. The processor 140 can control the encoder 120, and may also be able to control the HDMI interface 110 and / or the communication interface 130. The processor 140 can perform various types of control and information processing by executing software programs stored in the memory 150.

[0047] Specifically, the processor 140 can obtain an index of the visibility of the difference in image quality between blocks in the frame to be coded. In this embodiment, the index of the visibility of the difference in image quality between blocks in the frame to be coded is the quantization parameter of the frame to be coded.

[0048] The quantization parameter is a parameter that controls the size of the quantization step when quantizing a prediction error signal. Increasing the value of the quantization parameter increases the size of the quantization step, thereby reducing the bit rate, but also increasing the amount of information lost from the prediction error signal to be quantized (quantization distortion). Because intra-coding and inter-coding use different methods for generating prediction error signals, increasing quantization distortion increases the visibility of the difference in image quality between intra-coded blocks and inter-coded blocks. In other words, increasing the value of the quantization parameter increases the size of the quantization step, thereby increasing the visibility of the difference in image quality between blocks due to intra-refresh. Conversely, decreasing the value of the quantization parameter decreases the size of the quantization step, thereby reducing the visibility of the difference in image quality between blocks due to intra-refresh.

[0049] The quantization parameter of the current frame to be coded means a quantization parameter determined for the current frame to be coded. The quantization parameter of the current frame to be coded may be adjustable in units of slices, tiles, blocks, or transform coefficients within a block within the current frame to be coded.

[0050] Next, the processor 140 can determine whether to apply intra-refresh to the frame to be coded based on the obtained indicator. For example, the processor 140 may compare the value of the quantization parameter with a first threshold and determine whether to apply intra-refresh to the frame to be coded based on the comparison result.

[0051] Specifically, the processor 140 may determine to apply intra-refresh to the frame to be coded when the value of the quantization parameter is less than a first threshold. This allows the processor 140 to determine to apply intra-refresh to the frame to be coded when the visibility of differences in image quality between blocks is relatively low. Conversely, the processor 140 may determine not to apply intra-refresh to the frame to be coded when the value of the quantization parameter is equal to or greater than the first threshold. This allows the processor 140 to determine not to apply intra-refresh to the frame to be coded when the visibility of differences in image quality between blocks is relatively high.

[0052] The first threshold may be a quantization parameter value indicating that the visibility of the difference in image quality between blocks is sufficiently low to apply intra-refresh. The first threshold may be experimentally and / or empirically determined in advance and stored in the memory 150. The first threshold may be fixed or may be variable depending on the encoding parameters. For example, the first threshold may vary depending on the resolution of the frame to be encoded. Since the higher the resolution, the lower the visibility of the difference in image quality between blocks, the higher the first threshold may be. For example, the first threshold may be "34" for 4K resolution and "20" for 2K resolution.

[0053] Finally, the processor 140 may control the encoder 120 to encode the current frame based on the result of the determination of whether to apply intra-refresh to the current frame.

[0054] The memory 150 can store information used by the processor 140 to control the encoder 120. The memory 150 may be an electronic circuit or a collection of multiple electronic circuits. The memory 150 may be either a non-volatile memory or a volatile memory, or a combination of a non-volatile memory and a volatile memory. The memory 150 may be included in the processor 140.

[0055] [Internal Configuration of Decoding Apparatus 200] Next, a description will be given of the internal configuration of the decoding apparatus 200. The decoding apparatus 200 includes a communication interface 210, a decoder 220, a display panel 230, a processor 240, and a memory 250.

[0056] The communication interface 210 is an interface for wireless communication with the encoding device 100. For example, the communication interface 210 can receive communication data transmitted by the encoding device 100 and depacketize the received data to output encoded video data to the decoder 220. Note that the communication interface 210 may be an interface for wired communication with the encoding device 100, and may not be included in the decoding device 200.

[0057] The decoder 220 can decode the encoded video data. Specifically, the decoder 220 can decode a frame to be decoded included in the encoded video data in units of blocks.

[0058] The display panel 230 can display the video data decoded by the decoder 220. The display panel 230 may be, but is not limited to, a liquid crystal display panel or an organic electroluminescence (EL) display panel. Note that the display panel 230 does not necessarily have to be included in the decoding device 200.

[0059] The processor 240 is a circuit that processes information and has access to the memory 250. The processor 240 can control the decoder 220, and may also be able to control the communication interface 210 and / or the display panel 230.

[0060] The memory 250 can store information used by the processor 240 to control the decoder 220. The memory 250 may be an electronic circuit or a collection of multiple electronic circuits. The memory 250 may be either a non-volatile memory or a volatile memory, or a combination of a non-volatile memory and a volatile memory. The memory 250 may be included in the processor 240.

[0061] [Intra Refresh] Next, intra refresh will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of a video sequence for explaining intra refresh. Note that, hereinafter, frames to which intra refresh is applied are referred to as IR frames, and frames to which intra refresh is not applied are referred to as non-IR frames.

[0062] Here, the "period" refers to a period for intra-refresh. Specifically, the period refers to a repetition period of an IR frame sequence, or a repetition period of an IR frame and a non-IR frame sequence. In FIG. 3, the length of each period is 10 frames. In other words, if the frame rate is 60 Hz, one period is 1 / 6 second. Note that the length of the period may be fixed or variable.

[0063] 3 shows cycle 1 and cycle 2 included in a video sequence. Cycle 1 is a period in which intra-refresh is applied (hereinafter referred to as an IR cycle), and includes IR frames 10 to 14 and non-IR frames 15 to 19 in a predetermined order. Cycle 2 is a period in which intra-refresh is not applied (hereinafter referred to as a non-IR cycle), and includes non-IR frames 20 to 29, but does not include an IR frame.

[0064] IR frames 10 to 14 include intra regions (hatched regions in FIG. 3 ) at different positions. An intra region is a collection of blocks that are forcibly intra-coded, and in FIG. 3 represents one or more block lines. By distributing the intra regions at different positions across multiple frames, it is possible to prevent the amount of coding from concentrating on any one frame, and it is possible to provide a starting point for decoding (recovery point) even if image data from any region is lost due to packet loss. Areas other than the intra regions (non-hatched areas in FIG. 3 ) are collections of blocks for which intra-coding or inter-coding can be selected.

[0065] The shape and arrangement of the intra region are not limited to those shown in Figure 3. It is sufficient that a block at each position within a frame is included in the intra region at least once in any frame within one period. For example, the intra region may be one or more rows of blocks extending vertically.

[0066] Non-IR frames 15 to 29 include regions where intra-coding and inter-coding can be selected, and do not include intra-regions that are forced to be intra-coded.

[0067] The configuration of the IR cycle is not limited to the configuration shown in FIG. 3 . For example, the length of the cycle may be 60 frames. In this case, if the frame rate is 60 Hz, one cycle is 1 second. Also, for example, the ratio of the number of IR frames to non-IR frames included in an IR cycle is not limited to 1:1. For example, an IR cycle may include only IR frames. In other words, an IR cycle may not include non-IR frames. Also, for example, an IR cycle may not include multiple consecutive IR frames. In other words, an IR cycle may include multiple discrete IR frames. Also, for example, the first frame of an IR cycle may be a non-IR frame instead of an IR frame.

[0068] [Encoding Method] Next, the encoding method executed by the encoding device 100 will be described with reference to Figures 4 to 8. Figure 4 is a flowchart showing the encoding method according to this embodiment. Note that Figure 4 shows an exemplary flowchart of the encoding method, and the steps and the order of the steps of the encoding method are not limited to those described below.

[0069] The processor 140 obtains a quantization parameter for the current frame (S100). For example, the processor 140 determines the quantization parameter for the current frame based on a predetermined target code amount (target bit rate). In other words, the processor 140 determines the quantization parameter so that the target bit rate is satisfied. The current frame is the frame to be coded, and may also be called the picture to be coded or the current picture.

[0070] The processor 140 determines whether or not to set the current frame as an intraframe (S200). Details of step S200 will be described later with reference to FIG. 5. Note that step S200 is an optional step and does not necessarily need to be included in the encoding method according to this embodiment.

[0071] The processor 140 determines whether to apply intra-refresh (IR) to the current frame based on the quantization parameter (S300). That is, the processor 140 determines whether to set the current frame to an IR frame based on the quantization parameter. Details of step S300 will be described later using FIG. 6.

[0072] The encoder 120 encodes the current frame based on the result of the determination (S400). That is, the processor 140 controls the encoder 120 to encode the current frame based on the result of the determination. Details of step S400 will be described later with reference to FIG. 7.

[0073] [Details of Intra-Frame Determination Process] Details of the intra-frame determination process (step S200 in FIG. 4) will now be described with reference to FIG. 5. FIG. 5 is a flowchart showing the intra-frame determination process according to this embodiment. Note that FIG. 5 shows an exemplary flowchart of the intra-frame determination process, and the intra-frame determination process is not limited to the following description.

[0074] The processor 140 determines whether a packet loss notification has been received from the communication interface 130 (S202). A packet loss notification is a notification indicating that a packet loss has occurred in the communication of a frame prior to the current frame. When the communication interface 130 detects that the packet loss rate exceeds a certain value, it notifies the processor 140 of the packet loss along with the packet loss rate. The packet loss rate is the ratio of the number of packets not received by the communication interface 210 to the number of packets transmitted by the communication interface 130. The packet loss notification is notified based on the packet loss rate in the communication of multiple packets that make up the frame prior to the current frame.

[0075] If a packet loss notification has been received (Yes in S202), the processor 140 sets the current cycle including the current frame to a non-IR cycle (S204). That is, the processor 140 sets the current cycle to an intra-refresh off period. Note that there may be cases where the cycle is switched from an IR cycle to a non-IR cycle midway through.

[0076] The processor 140 determines whether the current state is a recovery waiting state (S206). The recovery waiting state refers to a state in which recovery from a packet loss has not been completed since the packet loss occurred. Note that the initial state at the start of a sequence of multiple frames including the current frame is set to a non-recovery waiting state.

[0077] If the current state is not a recovery waiting state (No in S206), the processor 140 sets the current state to a recovery waiting state (S210) and determines whether the content of the packet loss notification is serious (S212). Whether the content of the packet loss notification is serious can be determined by whether the packet loss notification satisfies a serious condition. Examples of serious conditions include packet loss occurring for a number of consecutive frames equal to or greater than a threshold, and / or a packet loss rate equal to or greater than a threshold. Note that serious conditions are not limited to these.

[0078] If the packet loss notification is not serious (No in S212), the processor 140 determines whether the recovery wait time is longer than a predetermined time (S214). The recovery wait time refers to the time during which the recovery wait state continues. A predetermined time, such as 20 seconds, can be used as the predetermined time.

[0079] If the recovery wait time is not longer than the predetermined time (No in S214), the processor 140 determines whether a scene change has occurred in the current frame (S216). The determination of whether a scene change has occurred is performed by determining whether the difference between the feature of the previous frame and the feature of the current frame is equal to or greater than a threshold. The feature may be, for example, a representative value (e.g., arithmetic mean or median) of pixel sample values ​​(luminance and / or color difference values), or the sum of absolute differences between sample values ​​of adjacent pixels. Furthermore, in the above determination, the result of motion estimation and / or the sum of prediction errors may be used instead of the feature.

[0080] If there is no scene change in the current frame (No in S216), the processor 140 ends the intraframe determination process. That is, the processor 140 skips step S208 and does not set the current frame as an intraframe.

[0081] If the current state is a recovery wait state (Yes in S206), if the packet loss notification is severe (Yes in S212), if the recovery wait time is longer than a predetermined time (Yes in S214), or if there is a scene change in the current frame (Yes in S216), the processor 140 sets the current frame to an intraframe (S208) and ends the intraframe determination process. Note that in step S208, the processor 140 sets the current frame to an intraframe, but it may also set a frame after the current frame to an intraframe. For example, if the current frame is not the first frame of the current cycle, the processor 140 may set the first frame of the next cycle to an intraframe.

[0082] If there is no packet loss notification (No in S202), the processor 140 determines whether the current state is a recovery wait state (S218). If the current state is not a recovery wait state (No in S218), the processor 140 ends the intraframe determination process. In other words, the processor 140 skips step S208 and does not set the current frame to an intraframe. If the current state is a recovery wait state (Yes in S218), the processor 140 executes the process of step S214.

[0083] [Details of Intra-Refresh Determination Process] Next, details of the intra-refresh determination process (step S300 in FIG. 4) will be described with reference to FIG. 6. FIG. 6 is a flowchart showing the intra-refresh determination process according to this embodiment. Note that FIG. 6 shows an exemplary flowchart of the intra-refresh determination process, and the intra-refresh determination process is not limited to the following description.

[0084] The processor 140 determines whether the current frame is set as an intraframe (S302). If the current frame is set as an intraframe (Yes in S302), the intrarefresh determination process ends.

[0085] If the current frame is not set as an intraframe (No in S302), the processor 140 determines whether the current frame is the first frame of the cycle (S304).If the current frame is the first frame of the cycle (Yes in S304), the processor 140 determines whether the value of the quantization parameter of the current frame is less than the first threshold (S306).

[0086] If the value of the quantization parameter of the current frame is less than the first threshold (Yes in S306), the processor 140 sets the current period to the IR period (S308). When the value of the quantization parameter is a small value less than the first threshold, the difference in image quality between intra-coded blocks and inter-coded blocks is less visible. In other words, even if intra-refresh is applied, the difference in image quality between the intra region and other regions is less visible. Therefore, by setting the current period to the IR period when the value of the quantization parameter is less than the first threshold, the processor 140 can apply intra-refresh while suppressing image quality degradation. The processor 140 sets the current frame to the IR frame (S310) and ends the intra-refresh determination process.

[0087] If the value of the quantization parameter of the current frame is not less than the first threshold (No in S306), the processor 140 sets the current period to a non-IR period (S312). When the value of the quantization parameter is a large value equal to or greater than the first threshold, the difference in image quality between intra-coded blocks and inter-coded blocks is easily visible. In other words, when intra-refresh is applied, the difference in image quality between intra-regions and other regions is easily visible. Therefore, by not setting the current period to an IR period when the value of the quantization parameter is equal to or greater than the first threshold, the processor 140 can suppress image quality degradation due to intra-refresh. The processor 140 sets the current frame to a non-IR frame (S314) and ends the intra-refresh determination process.

[0088] As described above, the processor 140 can determine whether or not to apply intra-refresh for each cycle by using the first frame of the cycle.

[0089] If the current frame is not the first frame of the cycle (No in S304), the processor 140 determines whether the current cycle is set to an IR cycle (S316). If the current cycle is set to an IR cycle (Yes in S316), the processor 140 determines whether the current frame is a frame that can be set as an IR frame within the IR cycle (S318). That is, the processor 140 determines whether the order of the current frame within the IR cycle (e.g., display order or encoding order) satisfies a predetermined condition. For example, in the example of FIG. 3, the processor 140 determines whether the display order of the current frame within the IR cycle is within the range of first to fifth (frames 10 to 14). If the current frame is a frame that can be set as an IR frame within the IR cycle (Yes in S318), the processor 140 sets the current frame to an IR frame (S320) and ends the intra-refresh determination process.

[0090] If the current period is not set to the IR period (No in S316), and if the current frame is not a frame that can be set to an IR frame within the IR period (No in S318), the processor 140 sets the current frame to a non-IR frame (S322) and terminates the intra-refresh determination process.

[0091] [Details of Frame Encoding Process] Next, details of the frame encoding process (step S400 in FIG. 4) will be described with reference to FIG. 7. FIG. 7 is a flowchart showing the frame encoding process according to this embodiment. Note that FIG. 7 shows an exemplary flowchart of the frame encoding process, and the frame encoding process is not limited to the following description.

[0092] The processor 140 determines whether the current frame is set as an intraframe (S402). If the current frame is set as an intraframe (Yes in S402), the processor 140 causes the encoder 120 to intracode the current frame (S404). Furthermore, the processor 140 sets the current state to a non-recovery waiting state (S406) and ends the frame encoding process.

[0093] If the current frame is not set as an intraframe (No in S402), the processor 140 determines whether the current frame is set as an IR frame (S408). If the current frame is set as an IR frame (Yes in S408), the processor 140 causes the encoder 120 to perform IR encoding of the current frame (S410). That is, the encoder 120 intra-codes a portion of the current frame based on the order of the current frame within the IR period. Thereafter, the processor 140 sets the current state to a non-recovery wait state (S406) and ends the frame encoding process.

[0094] If the current frame is not set as an IR frame (No in S408), the processor 140 causes the encoder 120 to non-IR-encode the current frame (S412), and ends the frame encoding process. In other words, the encoder 120 does not IR-encode the current frame.

[0095] [Specific Example of Encoding] A specific example of a video sequence encoded by the encoding method described above will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of a video sequence encoded by the encoding device 100 according to the embodiment.

[0096] Since there is no packet loss notification for frame 10 and frame 10 is the first frame of period 1, it is determined whether the value of the quantization parameter for frame 10 is less than the first threshold. Here, since the value of the quantization parameter for frame 10 is less than the first threshold, period 1 is set to an IR period and frame 10 is set to an IR frame. As a result, frame 10 is IR-encoded.

[0097] There is no packet loss notification for frames 11 to 17 either, and the current state is a non-recovery waiting state. Furthermore, period 1 is set to an IR period. Therefore, according to the order of frames 11 to 17 within period 1, frames 11 to 14 are set to IR frames, and frames 15 to 17 are set to non-IR frames. As a result, frames 11 to 14 are IR-encoded, and frames 15 to 17 are non-IR-encoded.

[0098] In frame 18, because there is a packet loss notification, period 1 is changed to a non-IR period. Furthermore, the current state is changed from a non-recovery waiting state to a recovery waiting state. Here, the packet loss notification is not severe, the recovery waiting time is shorter than the predetermined time, and there is no scene change. Therefore, frame 18 is not set to an intra frame and is not intra-coded. Furthermore, because frame 18 is included in a non-IR period, it is not set to an IR frame and is not IR-coded.

[0099] In frame 19, there is no packet loss notification and the current state is a recovery waiting state. Furthermore, the recovery waiting time is shorter than a predetermined time and there is no scene change. Therefore, frame 19 is not set as an intra frame and is not intra coded.

[0100] Because the value of the quantization parameter for frame 20, the first frame of cycle 2, is equal to or greater than the first threshold, cycle 2 is set to a non-IR cycle. For frames 20 to 23 included in cycle 2, which is set to a non-IR cycle, there is no packet loss notification and the current state is a recovery wait state. Furthermore, the recovery wait time is shorter than a predetermined time and there is no scene change. Therefore, frames 20 to 23 are not set to IR frames and are not IR-coded. As a result, the recovery wait state is maintained.

[0101] In frame 24, there is no packet loss notification, and the current state is a recovery waiting state. Furthermore, although the recovery waiting time is shorter than the predetermined time, there is a scene change from the first scene to the second scene. Therefore, frame 24 is set to an intra frame and intra-coded. As a result, the current state is changed from a recovery waiting state to a non-recovery waiting state.

[0102] For frames 25 to 29, there is no packet loss notification and the current state is a non-recovery waiting state. Furthermore, period 2 is set to a non-IR period. Therefore, frames 25 to 29 are set to non-IR frames and are non-IR encoded.

[0103] (Modifications) While the encoding device and encoding method have been described above based on the embodiments, the encoding device and encoding method according to the present disclosure are not limited to the above embodiments. The present disclosure also includes other embodiments realized by combining any of the components in the above embodiments, and modifications obtained by applying various modifications to the above embodiments that would occur to those skilled in the art without departing from the spirit of the present disclosure.

[0104] For example, in the above embodiment, the value of the quantization parameter is used as an indicator of the visibility of the difference in image quality between blocks in the current frame to be coded. However, this is not limiting. For example, the value of the skip mode selection rate in the frame preceding the current frame to be coded may be used as an indicator of the difference in image quality between blocks. The value of the skip mode selection rate is the ratio of the number of blocks to which skip mode is applied to the total number of blocks included in the frame preceding the current frame to be coded. As the value of this skip mode selection rate increases, the number of blocks in skip mode increases, and the visibility of the difference in image quality between blocks increases. Conversely, as the value of the skip mode selection rate decreases, the number of blocks in skip mode decreases, and the visibility of the difference in image quality between blocks decreases. Therefore, the processor 140 may determine not to apply intra-refresh to the current frame to be coded if the value of the skip mode selection rate is equal to or greater than a second threshold, and may determine to apply intra-refresh to the current frame to be coded if the value of the skip mode selection rate is less than the second threshold. That is, step S306 in FIG. 6 may be replaced with a step of determining whether the value of the skip mode selection rate is less than the second threshold.

[0105] The second threshold may be a skip mode selection rate value indicating that the visibility of the difference in image quality between blocks is low enough to warrant applying intra-refresh. The second threshold may be experimentally and / or empirically determined in advance and stored in memory 150. Alternatively, the second threshold may be adaptively determined according to coding parameters, and the determined second threshold may be coded in the header.

[0106] For example, the intra-mode selection rate value in the frame preceding the current frame to be coded may be used as an indicator of the difference in image quality between blocks. The intra-mode selection rate value is the ratio of the number of blocks to which intra-mode is applied to the total number of blocks included in the frame preceding the current frame to be coded. As the intra-mode selection rate value increases, the number of intra-mode blocks increases, and the visibility of the difference in image quality between blocks due to intra-refresh becomes lower. Conversely, as the intra-mode selection rate value decreases, the number of intra-mode blocks decreases, and the visibility of the difference in image quality between blocks due to intra-refresh becomes higher. Therefore, the processor 140 may determine not to apply intra-refresh to the current frame to be coded if the intra-mode selection rate value is less than a third threshold, and may determine to apply intra-refresh to the current frame to be coded if the intra-mode selection rate value is equal to or greater than the third threshold. That is, step S306 in FIG. 6 may be replaced with a step of determining whether the intra-mode selection rate value is equal to or greater than the third threshold.

[0107] The third threshold may be a value of the intra-mode selection rate indicating that the visibility of the difference in image quality between blocks is low enough to apply intra-refresh. The third threshold may be predetermined experimentally and / or empirically and stored in the memory 150. Alternatively, the third threshold may be adaptively determined according to the encoding parameters, and the determined third threshold may be encoded in the header.

[0108] For example, the value of the amount of change between the current frame and the previous frame may be used as an index of the difference in image quality between blocks. The value of the amount of change indicates the amount of change between the current frame and the previous frame. The value of the amount of change may be the sum of the magnitudes of the motion vectors of each block of the current frame relative to the previous frame. The value of the amount of change may also be the ratio of the number of blocks having motion vectors with magnitudes equal to or greater than a threshold to the total number of blocks in the current frame. In these cases, the value of the amount of change increases when the motion between frames is large, and decreases when the motion between frames is small. The value of the amount of change may also be the sum of the absolute values ​​of the differences between each pixel value of the current frame and the corresponding pixel value of the previous frame. In this case, the value of the amount of change increases when the image change between frames is large, and decreases when the image change between frames is small. When the value of the amount of change between frames is large, the human eye has difficulty recognizing details on the screen, reducing the visibility of image quality differences between blocks due to intra-refresh. Conversely, when the value of the inter-frame change amount is small, the human eye can more easily recognize details on the screen, and the visibility of the image quality difference between blocks due to intra-refresh becomes higher. Therefore, the processor 140 may determine not to apply intra-refresh to the encoding target frame when the value of the change amount is less than the fourth threshold, and may determine to apply intra-refresh to the encoding target frame when the value of the change amount is equal to or greater than the fourth threshold. In other words, step S306 in FIG. 6 may be replaced with a step of determining whether the value of the change amount is equal to or greater than the fourth threshold.

[0109] The fourth threshold may be a value of the amount of change indicating that the visibility of the difference in image quality between blocks is low enough to warrant applying intra-refresh. The fourth threshold may be experimentally and / or empirically determined in advance and stored in the memory 150. Alternatively, the fourth threshold may be adaptively determined according to encoding parameters, and the determined fourth threshold may be encoded in the header.

[0110] In the above embodiment, the frame to be coded for which the determination of whether to apply intra-refresh based on the index of visibility of image quality differences between blocks is made is the first frame of a cycle for intra-refresh, but this is not limiting. For example, if the first frame of a cycle is a non-IR frame, the determination may be made for a frame in the cycle that corresponds to the first IR frame instead of the first frame.

[0111] The present disclosure is applicable to, for example, television receivers, digital video recorders, car navigation systems, mobile phones, digital cameras, digital video cameras, and the like.

[0112] 100 Encoding device 110 HDMI interface 120 Encoder 130, 210 Communication interface 140, 240 Processor 150, 250 Memory 200 Decoding device 220 Decoder 1000 Encoding / decoding system

Claims

1. An encoding device comprising: a processor; and an encoder that encodes a frame to be encoded on a block-by-block basis, wherein the processor obtains an index of visibility of image quality differences between blocks in the frame to be encoded; determines whether to apply intra-refresh to the frame to be encoded based on the obtained index; and controls the encoder to encode the frame to be encoded based on the result of the determination.

2. The encoding device of claim 1, wherein the processor obtains the value of a quantization parameter of the frame to be encoded as the index, determines to apply the intra-refresh to the frame to be encoded if the value of the quantization parameter is less than a first threshold, and determines not to apply the intra-refresh to the frame to be encoded if the value of the quantization parameter is greater than or equal to the first threshold.

3. The encoding device of claim 1, wherein the processor obtains as the index the value of a skip mode selection rate in a frame preceding the frame to be encoded, and determines not to apply the intra-refresh to the frame to be encoded if the value of the skip mode selection rate is greater than or equal to a second threshold, and determines to apply the intra-refresh to the frame to be encoded if the value of the skip mode selection rate is less than the second threshold.

4. The encoding device of claim 1, wherein the processor obtains the value of the intra-mode selection rate in the frame preceding the frame to be encoded as the index, determines to apply the intra-refresh to the frame to be encoded if the value of the intra-mode selection rate is greater than or equal to a third threshold, and determines not to apply the intra-refresh to the frame to be encoded if the value of the intra-mode selection rate is less than the third threshold.

5. The encoding device of claim 1, wherein the processor obtains the value of the amount of change between the frame to be encoded and the frame preceding the frame to be encoded as the index, determines to apply the intra-refresh to the frame to be encoded if the value of the amount of change is greater than or equal to a fourth threshold, and determines not to apply the intra-refresh to the frame to be encoded if the value of the amount of change is less than the fourth threshold.

6. The encoding device according to any one of claims 1 to 5, wherein the processor determines whether or not to apply intra-refresh in units of a cycle for intra-refresh including the frame to be encoded.

7. The encoding device according to claim 6, wherein the encoding target frame is a first frame in a cycle for intra-refresh.

8. The encoding device according to any one of claims 1 to 5, further comprising a communication interface for transmitting a plurality of encoding frames including the encoded encoding target frame via packet communication, and the processor further determines whether a waiting time for recovery is longer than a predetermined time when the encoding target frame is waiting for recovery due to packet loss in communication of a frame prior to the encoding target frame, and sets the encoding target frame to an intra frame in response to determining that the waiting time is longer than the predetermined time.

9. An encoding method for encoding a frame to be encoded on a block-by-block basis, comprising: obtaining an index of visibility of image quality differences between blocks in the frame to be encoded; determining whether or not to apply intra-refresh to the frame to be encoded based on the obtained index; and encoding the frame to be encoded based on a result of the determination.

10. A program for causing a computer to execute the encoding method according to claim 9.

Images