Estimation of weighted prediction parameters

Abstract

A method for estimating weighted prediction parameters intended to be used to predict image blocks is provided.

Description

Technology Field

[0001] The technical field of one or more embodiments generally relates to video compression. In particular, at least some embodiments relate to estimating weighted-prediction parameters.

[0002] At least some embodiments relate to improving compression efficiency compared to existing video compression systems such as HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2 described in "ITU-T H.265 Telecommunication standardization sector of ITU (10 / 2014), series H: audiovisual and multimedia systems, infrastructure of audiovisual services - coding of moving video, High efficiency video coding, Recommendation ITU-T H.265") or undeveloped video compression systems such as VVC (Versatile Video Coding (Draft 7), October 2019, Geneva, Document JVET-P2001, by B. Bross, J. Chen, S. Liu, YK. Wang).

[0003] Video compression systems provide rules for compressing and encoding data regarding images in a sequence of images. These rules provide a method of compressing and encoding image data to transmit less data than the original viewing camera provided around the images. Subsequently, this reduced volume of data requires less channel bandwidth for transmission. If a receiver knows the rules that the transmitter used to perform compression and encoding, the receiver can reconstruct (or decode) the images of the image sequence from the transmitted data.

[0004] Video compression systems encode image blocks using either an inter-prediction mode, an intra-prediction mode, or a combination of inter-intra-prediction modes. In the case of an inter-prediction mode, motion estimation / compensation is used to predict an image block from a reference image. The predicted block can be obtained, for example, by searching for the best matching image block in the reference image using a predetermined search window based on a predetermined matching error criterion. Subsequently, the matched image block may be an inter-prediction block used to predict the original image block, and the prediction residual block is further processed and sent to the decoder. The relative shifts of the reference image block in the horizontal and vertical directions relative to the original image block are grouped and referred to as the motion vectors of the original image block, which are also sent to the decoder.

[0005] In the case of intra prediction mode, image blocks within an image are predicted using spatial prediction methods. Intra prediction blocks can be obtained, for example, by searching for the best matching image block in the image with a predetermined search window based on a predetermined matching error criterion. Subsequently, the matched image block may be an intra prediction block used to predict the original image block, and the prediction residual block is further processed and transmitted to a decoder.

[0006] The primary objective of intra and inter prediction is to predict an image block such that the prediction residual block, obtained by taking the difference between the prediction block and the original image block, generates the minimum number of bits during encoding.

[0007] One of the tasks is to improve the prediction of image blocks to reduce the dynamic characteristics of the prediction residual blocks.

[0008] This is addressed by alternative embodiments described herein, which relate to estimating weighted prediction parameters.

[0009] According to the first aspect, a method for predicting image blocks using weighted prediction parameters is provided, wherein the weighted prediction parameters are estimated by the steps of: obtaining a first estimate of the weighted prediction parameters; obtaining a second estimate of the weighted prediction parameters based on the first estimate of the weighted prediction parameters and a scaled reference image histogram derived from samples of a reference image; and enabling the use of the second estimate of the weighted prediction parameters to predict image blocks based on an image histogram derived from samples of an image and a reference image histogram derived from samples of a reference image, and another histogram distortion calculated between the second estimate of the weighted prediction parameters and a scaled reference image histogram derived from the reference image histogram and an image histogram.

[0010] In one embodiment, the step of obtaining a second estimate of weighted prediction parameters depends on the configuration parameters.

[0011] In one embodiment, the step of obtaining a second estimate of weighted prediction parameters depends on the components of the image.

[0012] In a modified example, the steps of obtaining a second estimate of the weighted prediction parameters and enabling the use of the second estimate of the weighted prediction parameters to predict image blocks are performed for a first component of the image. The method further comprises, for a second component of the image, the step of enabling the use of the weighted prediction parameters to predict image blocks based on the first estimate of the weighted parameters, the sum of absolute differences between samples of the image and samples of a scaled reference image, and the sum of absolute differences between samples of the image and samples of the reference image.

[0013] In a modified example, the present method further includes the step of spreading an image histogram before obtaining a first estimate of weighted prediction parameters.

[0014] In one embodiment, the step of spreading the image histogram depends on the bit depth of the samples of the image and a given bit depth.

[0015] In one embodiment, the step of spreading the image histogram includes the step of distributing samples of the image around the peak of the image histogram with respect to the peak.

[0016] In one embodiment, the first estimate of the weighted prediction parameters is obtained from default weighted parameters or based on samples of the image and samples of the reference image.

[0017] According to other suns, a device, a non-transient computer-readable medium, and a computer program product are provided.

[0018] These and other suns, features, and advantages of alternative suns will become apparent from the following detailed description of exemplary embodiments to be read in conjunction with the attached drawings. Brief explanation of the drawing

[0019] In the drawings, examples of various embodiments are illustrated. The drawings illustrate the following: FIG. 1 illustrates a block diagram of an example of an encoder in which various suns and embodiments are implemented. FIG. 2 illustrates a block diagram of a video decoder in which various suns and embodiments are implemented. FIG. 3 illustrates a block diagram of an example of a system in which various suns and embodiments are implemented. FIG. 4 illustrates a flowchart of a method for estimating weighted prediction parameters according to at least one embodiment. FIG. 5 illustrates a flowchart of step (420) according to at least one embodiment. Figure 6 illustrates a typical example showing how an increase in bit depth from an 8-bit picture to a 10-bit picture affects histograms. FIG. 7 illustrates an example of histogram diffusion according to at least one embodiment. Specific details for implementing the invention

[0020] This application describes various aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity, and at least, in order to illustrate individual characteristics, are often described in a manner that may sound like a limitation. However, this is for the purpose of clarity in description and does not limit the application or scope of these aspects. In practice, all different aspects may be combined and interchanged to provide additional aspects. Furthermore, aspects may be combined and likewise interchanged with aspects described in prior applications.

[0021] The embodiments described and considered in this application may be implemented in many different forms. FIGS. 1, 2 and 3 provide some embodiments, but other embodiments are considered, and the discussion of FIGS. 1, 2 and 3 does not limit the scope of embodiments.

[0022] At least one of the embodiments relates generally to video encoding and decoding. These and other embodiments may be embodied as a method, an apparatus, a computer-readable medium storing instructions for encoding or decoding video data according to any of the aforementioned methods, or a computer program product comprising instructions that cause a computer to perform any of the aforementioned methods when the program is executed by a computer.

[0023] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Generally, but not necessarily, the term "reconstructed" is used on the encoder side, whereas "decoded" is used on the decoder side.

[0024] The term "histogram" refers to a histogram of sample values ​​or a cumulative histogram.

[0025] The term "histogram distortion" is also used. The histogram distortion "HD" between two histograms H1 and H2 can be defined by the sum of the absolute differences of the histogram values ​​of H1 and H2 associated with each bin, or the sum of the squared differences of the histogram values ​​of H1 and H2 associated with each bin "b".

[0026]

[0027] The histogram associated with an image component consists of N bins corresponding to N possible values ​​(known as range values) that image samples can have for this component. For example, an 8-bit image consists of three components (e.g., Y, U, V) that have values ​​between 0 and 255 (Y) or between -128 and 127 (U, V). The histogram value associated with bin "b" is the number of samples in the image that have the same value as "b" for this component. The cumulative histogram value associated with bin "b" is the number of samples in the image that have the same value as or lower than "b" for this component.

[0028] Various methods are described herein, and each of the methods comprises one or more steps or actions to achieve the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0029] Various methods and other embodiments described in this application may be used to modify modules, for example, a module (102) that derives coding parameters of the video encoder (100) of FIG. 1. Furthermore, these embodiments are not limited to specific video compression systems such as VVC or HEVC, and may be applied to other standards and recommendations, whether existing or future to be developed, and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the embodiments described in this application may be used individually or in combination.

[0030] Various numerical values, for example, the number of intra-prediction modes, are used in this application. Specific values ​​are for exemplary purposes and the described embodiments are not limited to these specific values.

[0031] FIG. 1 illustrates a block diagram of an encoder (100). Variations of this encoder (100) are considered, but the encoder (100) is described below for the sake of clarity without describing all expected variations.

[0032] Before encoding, the video sequence may undergo pre-encoding processing (101), for example, applying a color conversion to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components (e.g., using histogram equalization of one of the color components) to obtain a signal distribution that is more resilient to compression. Metadata may be associated with the pre-encoding processing (101) and may be attached to the bitstream.

[0033] In the encoder (100), the image is encoded by encoder elements as described below. The image to be encoded is partitioned (102) and processed into units of, for example, CUs (Coding Units). Each unit (typically, an image block) is encoded using, for example, either an inter-prediction mode or an intra-prediction mode, or a combined inter-intra-prediction mode. When a unit is encoded in an intra-prediction mode, it performs intra-prediction (160). In an inter-prediction mode, motion estimation (175) and compensation (170) are performed. In a combined inter-intra-prediction mode, it performs both intra-prediction (160) and motion estimation (175) and compensation (170). The encoder determines (105) one mode to use to encode the unit among the intra-prediction mode, the inter-prediction mode, or the combined inter-intra-prediction mode, and indicates the determination, for example, by a prediction mode flag. Predicted residuals are calculated, for example, by subtracting the predicted unit from the original unit (110).

[0034] Next, the predicted residuals are transformed (125) and quantized (130). The quantized transformation coefficients, as well as motion vectors and other syntax elements, are entropy-coded (145) to output a bitstream. The encoder can skip the transformation and apply quantization directly to the untransformed residual signal. The encoder can bypass both transformation and quantization, that is, the predicted residuals are coded directly without the application of transformation or quantization processes.

[0035] The encoder decodes the encoded unit to provide a reference for further predictions. The quantized transform coefficients are inversely quantized (140) and inversely transformed (150) to decode the prediction residuals. The unit is reconstructed by combining the decoded prediction residuals and the predicted unit (155). In-loop filters (165) are applied to the reconstructed unit to perform image de-blocking or SAO (Sample Adaptive Offset) filtering, for example, to reduce encoding artifacts. The filtered unit is stored in a reference picture buffer (known as the decoded picture buffer (DPB)) (180).

[0036] FIG. 2 illustrates a block diagram of a video decoder (200). Variations of this decoder (200) are considered, but the decoder (200) is described later for the sake of clarity without describing all expected variations.

[0037] In the decoder (200), the bitstream is decoded by decoder elements as described below.

[0038] The video decoder (200) generally performs a decoding pass that is reciprocal to the encoding pass as described in FIG. 2. The decoder (200) also generally performs video decoding as part of encoding video data.

[0039] In particular, the input to the decoder includes a video bitstream that can be generated by the video encoder (100). The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition the image (235) according to the decoded picture partitioning information. The transform coefficients are inversely quantized (240) and inversely transformed (250) to decode the prediction residuals. By combining the decoded prediction residuals and the predicted unit (255), a unit (typically, an image block) is reconstructed. The predicted unit may be obtained (270) from an intra prediction (260) or a motion-compensated prediction (i.e., an inter prediction) (275), or from both the intra prediction (260) and the motion-compensated prediction (i.e., an inter prediction) (275). In-loop filters (265) are applied to the reconfigured unit. The filtered unit is stored in the reference picture buffer (280).

[0040] The decoded image may additionally undergo post-decoding processing (285), for example, inverse color conversion (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4), or inverse remapping, which performs the reverse of the remapping process performed in pre-encoding processing (101). The post-decoding processing may use metadata derived from pre-encoding processing and signaled to the bitstream.

[0041] FIG. 3 illustrates a block diagram of an example of a system in which various embodiments and embodiments are implemented. The system (3000) may be implemented as a device comprising various components described below and configured to perform one or more of the embodiments described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital TV receivers, personal video recording systems, connected home appliances, and servers. The elements of the system (3000) may be implemented alone or in combination on a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of the system (3000) are distributed across multiple ICs and / or separate components. In various embodiments, the system (3000) is communically coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, the system (3000) is configured to implement one or more of the embodiments described in this document.

[0042] The system (3000) includes at least one processor (3010) configured to execute instructions loaded internally to implement, for example, the various aspects described in this document. The processor (3010) may include embedded memory, an input / output interface, and various other circuit components as known in the art. The system (3000) includes at least one memory (3020) (e.g., a volatile memory device and / or a non-volatile memory device). The system (3000) includes a storage device (3040) that may include non-volatile memory and / or volatile memory, including but not limited to electrically eraseable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash, magnetic disk drive and / or optical disk drive. The storage device (3040) may include, as non-limiting examples, an internal storage device, a combined storage device (including a removable storage device and a non-removable storage device), and / or a network-accessible storage device.

[0043] The system (3000) includes, for example, an encoder / decoder module (3030) configured to process data to provide an encoded video or a decoded video, and the encoder / decoder module (3030) may include its own processor and memory. The encoder / decoder module (3030) represents a module(s) that may be included in a device to perform encoding and / or decoding functions. As is known, the device may include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module (3030) may be implemented as a separate element of the system (3000) or may be integrated within the processor (3010) as a combination of hardware and software as known to those skilled in the art.

[0044] Program code to be loaded onto a processor (3010) or an encoder / decoder (3030) to perform the various aspects described in this document may be stored in a storage device (340) and subsequently loaded onto memory (3020) for execution by the processor (3010). According to various embodiments, one or more of the processor (3010), memory (3020), storage device (3040), and encoder / decoder module (3030) may store one or more of the various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or parts of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0045] In some embodiments, memory within the processor (3010) and / or encoder / decoder module (3030) is used to store instructions and to provide working memory for processing required during encoding or decoding.

[0046] However, in other embodiments, memory outside the processing device (e.g., the processing device may be either a processor (3010) or an encoder / decoder module (3030)) is used for one or more of these functions. The external memory may be memory (3020) and / or a storage device (3040), e.g., dynamic volatile memory and / or non-volatile flash memory.

[0047] In some embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television.

[0048] In at least one embodiment, high-speed external dynamic volatile memory, such as RAM, is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by the Joint Video Experts Team (JVET)).

[0049] Inputs to the elements of the system (3000) may be provided through various input devices as shown in the module (3130). Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives a radio frequency (RF) signal transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples not shown in FIG. 3 include composite video.

[0050] In various embodiments, the input devices of the module (3130) are associated with their respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting the signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting it again to a narrower band of frequencies to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF portion of various embodiments includes one or more elements for performing these functions, for example, frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various such functions, including, for example, down-converting the received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to the baseband.

[0051] In one set-top box embodiment, the RF part and its associated input processing element receive an RF signal transmitted through a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering for a desired frequency band.

[0052] Various embodiments rearrange the order of the aforementioned (and other) elements, remove some of these elements, or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters.

[0053] In various embodiments, the RF portion includes an antenna.

[0054] Additionally, the USB and / or HDMI terminals may each include an interface processor for connecting the system (3000) to other electronic devices through the USB and / or HDMI connections.

[0055] It should be understood that various forms of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, within a separate input processing IC or within the processor (3010).

[0056] Similarly, the suns of USB or HDMI interface processing can be implemented in separate interface ICs or in the processor (3010) as needed.

[0057] The demodulation, error correction, and demultiplexed streams are provided to various processing elements, including a processor (3010) and an encoder / decoder (3030), which operate in combination with memory and storage elements to process the data stream as needed, for example, for presentation on an output device.

[0058] Various elements of the system (3000) may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected using an internal bus as known in the art, which includes a suitable connection arrangement (12²140), e.g., an I2C bus, wiring, and printed circuit boards, and may transmit data between them.

[0059] The system (3000) includes a communication interface (3050) that enables communication with other devices through a communication channel (3060). The communication interface (3050) may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel (3060). The communication interface (3050) may include, but is not limited to, a modem or a network card, and the communication channel (3060) may be implemented, for example, within a wired and / or wireless medium.

[0060] In various embodiments, data is streamed to or otherwise provided to the system (3000) using a wireless network, such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). Wi-Fi signals in these embodiments are received through a communication channel (3060) and a communication interface (3050) adapted for Wi-Fi communication. The communication channel (3060) in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communication.

[0061] Other embodiments provide data streamed to the system (3000) using a set-top box that transmits data through the HDMI connection of the input image block (3130). Still other embodiments provide data streamed to the system (3000) using the RF connection of the input module (3130).

[0062] As described above, various embodiments provide data in a non-streaming manner.

[0063] In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0064] The system (3000) can provide output signals to various output devices including a display (3100), speakers (3110), and other peripheral devices (3120).

[0065] The display (3100) of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display (3100) may be for a television, tablet, laptop, cell phone (mobile phone), or other device. The display (3100) may also be integrated with other components (for example, as in a smartphone) or separate (for example, an external monitor for a laptop).

[0066] In various examples of the embodiments, other peripheral devices (3120) include one or more of a standalone digital video disc (or digital multifunction disc) (for both terms, DVR), a disc player, a stereo system, and / or a lighting system.

[0067] Various embodiments use one or more peripheral devices (3120) that provide functions based on the output of the system (3000). For example, a disc player performs the function of playing the output of the system (3000).

[0068] In various embodiments, control signals are communicated between a system (3000) and a display (3100), speakers (3110), or other peripheral devices (3120) using signaling such as AV.Link, CEC (Consumer Electronics Control), or other communication protocols that enable device-to-device control with or without user intervention.

[0069] Output devices can be coupled to the system (3000) via dedicated connections through their respective interfaces (3070, 3080, 3090) to enable communication.

[0070] Alternatively, output devices can be connected to the system (3000) using a communication channel (3060) through a communication interface (3050).

[0071] The display (3100) and speakers (3110) can be integrated into a single unit with other components of the system (3000) in an electronic device, for example, such as a television.

[0072] In various embodiments, the display interface (3070) includes a display driver, such as a timing controller (T Con) chip, for example.

[0073] Alternatively, the display (3100) and speaker (3110) may be separate from one or more of the other components, for example, if the RF portion of the input (3130) is part of a separate set-top box.

[0074] In various embodiments where the display (3100) and speakers (3110) are external components, the output signal may be provided through dedicated output connections, such as HDMI ports, USB ports, or COMP outputs.

[0075] The embodiments may be performed by a processor (3010), by computer software implemented by hardware, or by a combination of hardware and software.

[0076] As a non-limiting example, embodiments may be implemented by one or more integrated circuits.

[0077] The memory (3020) may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples.

[0078] The processor (3010) may be of any type suitable for the technical environment and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0079] In inter-prediction mode, image block cur can be predicted from motion-compensated samples of reference block ref of a reference picture stored in a previously reconstructed (decoded) decoded picture buffer (DPB). Next, the motion-compensated samples (also referred to as inter-prediction samples) can be corrected with weighted prediction parameters (WP parameters) to compensate for lighting changes between the samples of image block cur and the motion-compensated samples of reference block ref.

[0080] For example, weighted-predictions (WP) as defined in HEVC or VVC are weights and offsets (w) associated with a reference picture "i", per group of units, e.g., per group of blocks, per region, per slice, or per entire picture. i ; off i It allows defining ). In the decoded picture buffer (DPB), each block group (e.g., reference image) is associated with weighted-predict parameters. In the following, the term image will be used, but it may be replaced with a slice or region within the image, where a region is typically a block group.

[0081] Next, WP is applied to inter-predicted samples.

[0082] WP can be applied to both unidirectional and bidirectional predictions.

[0083] For example, if image block cur is unidirectionally coded with reference block index refIdx0, (w0;off0) can refer to the weight and offset associated with refIdx0. Subsequently, weighted unidirectional prediction samples Pred'0 and weighted bidirectional prediction samples Pred' 01 It can be given by the following:

[0084] Pred'0=((w0.Pred0[x]+(1<<(shift-1)))>>shift)+off0(1)

[0085] Pred' 01 =((w0.Pred0[x]+w1.Pred1[x]+off 01 )>>(shift+1) (2)

[0086] Off here 01 = (off0 + off1 + 1) << shift, and Pred0, located at position x in the current image, are unidirectional prediction samples and Pred 01 is a bidirectional prediction sample.

[0087] FIG. 4 illustrates a flowchart of a method for estimating weighted prediction parameters according to at least one embodiment.

[0088] In step (410), a first estimate of the WP parameters is obtained.

[0089] In one embodiment of step (410), a first estimate of the WP parameters is obtained from default parameters and possibly received or stored (the weight may be equal to 1 and the offset may be equal to 0).

[0090] In one embodiment of step (410), a first estimate of the WP parameters is obtained based on samples of image cur and (possibly co-located) samples of reference image ref.

[0091] In one embodiment of step (410), the first estimate of the WP parameters is the average sample value (DC) calculated for image cur and reference image ref, respectively. cur , DC ref ) and mean standard deviation or mean of absolute differences (AC cur , AC ref It is obtained based on ).

[0092] Next, the weights and offsets of the WP parameters can be derived as follows:

[0093] weight = AC cur / AC ref

[0094] offset = DC cur - weight × DC ref

[0095] For example, the average sample value DC for component "C" of image "cur" cur This is given by the following:

[0096]

[0097] Here, "cur(x)" is the sample value at position "x" within the image "cur" for component "C".

[0098] For example, the average value of the absolute differences AC cur is given by the following:

[0099]

[0100] In step (420), the second estimate of the WP parameters is the first estimate of the WP parameters, the scaled reference image histogram H' derived from samples of the current image cur and (possibly co-located) samples of the reference image ref. scaled It is based on.

[0101] FIG. 5 illustrates a flowchart of step (420) according to at least one embodiment.

[0102] In step (4201), the image histogram H of the samples of image cur cur and reference image histogram H of (possibly co-located) samples of the reference image ref ref is calculated.

[0103] In step (4202), the current WP parameters (weight; offset) are derived from the previous WP parameter values ​​by, for example, adding a given value to the previous weight and offset values.

[0104] In step (4203), the scaled reference image histogram H scaled is the reference image histogram H as follows ref and is derived from the current WP parameters (weight; offset):

[0105]

[0106] At this time, i is H ref It is empty.

[0107] In step (4204), histogram distortion D(H cur , H scaled) is image histogram H cur and scaled reference image histogram H scaled It is calculated in between.

[0108] The current WP parameters are, D(H cur , H scaled If ) < min(value of histogram distortion associated with previously stored WP parameters), it is saved (selected). Otherwise, new current WP parameters are considered in step (4202), and steps (4203 to 4204 (4205)) are repeated.

[0109] Returning to Fig. 4, at step (430), the histogram distortion "HD_noWP"=D(H cur , H ref ) is image histogram H cur and reference image histogram H ref It is calculated between. Other histogram distortions "HD_WP"=D(H cur , H' scaled ) is the scaled reference image histogram H' derived from step (4203) when the stored WP parameters are used. scaled and image histogram H cur It is calculated in between.

[0110] Next, D(H curr , H' scaled ) < D(H curr , H ref In the case where ), a second estimate of the WP parameters is used to predict image blocks (enabled) (440). Otherwise (D(H curr , H' scaled ) >= D(H curr , H ref )), the second estimate of the WP parameters is disabled (not used to predict image blocks) (450).

[0111] In a variation, the information indicates whether the first estimate of the weighted parameters has been refined (step (440 or 450) following steps (420, 430)) or not refined (step (440 or 450) following step (460)).

[0112] In a variation example, when the first estimate of the WP parameters is obtained from the default parameters (where the weight may be equal to 1 and the offset may be equal to 0), steps (step (440 or 450) following steps (420, 430)) follow step (410).

[0113] In a variation, this information may be a configuration parameter of the encoder.

[0114] In a variation, the information relates to a component of an image. Subsequently, steps (420 and 430) may be performed for a component of a picture, and step (460) may be performed for another component of the picture.

[0115] In the modified example, the same information is used for all components of the image. Subsequently, either step (420 and 430) or step (460) may be executed for these components.

[0116] In step (460), the sum of the absolute differences (SAD_WP) is calculated between the samples of image cur and the (possibly co-located) samples of the scaled reference image.

[0117]

[0118] Another sum of absolute differences (SAD_noWP) is calculated between samples of image cur and (possibly co-located) samples of reference image ref.

[0119]

[0120] Next, SAD_WP <SAD_noWP인 경우, 이미지 블록들을 예측하기 위해 WP 파라미터들의 제1 추정치가 사용된다(인에이블됨)(440). 그렇지 않은 경우(SAD_WP > =SAD_noWP), the first estimate of the WP parameters is disabled (not used to predict image blocks) (450).

[0121] The use of SAD to enable or disable the use of WP parameters to predict image blocks assumes that the video sequence contains little motion, because SAD measures the difference between samples (possibly co-located or globally motion-compensated) within the image and reference image. However, for videos with sequences containing moving objects, the validity of SAD is significantly reduced. This problem can be resolved by using histogram-based distortions (step (430)) to make better decisions regarding enabling or disabling the estimated WP parameters to predict image blocks.

[0122] In a variation, the present method may include a step of increasing the bit depth of samples of image blocks to be predicted. Subsequently, internal samples having an increased bit depth relative to the bit depth of samples of image blocks are acquired and stored. This increase in bit depth increases numerical precision in sample-based prediction processes.

[0123] For example, 8-bit samples of an image can be increased to 10-bit internal samples.

[0124] The image histogram and reference image histogram can be significantly affected by the bit-depth increase as shown in Fig. 6.

[0125] The top of Fig. 6 illustrates a typical example showing how an increase in bit depth from an 8-bit picture to a 10-bit picture affects histograms. The top-left histogram represents a portion of the 8-bit image histogram, and the top-right is the 10-bit image histogram (left-shifted by 2 bits). It can be seen that the increase in bit depth generates empty histogram values ​​for some bins. However, loss coding algorithms will not reconstruct the same sample values ​​exactly, and the histogram of the reconstructed sample values ​​may look very different, as exemplified at the bottom of Fig. 6.

[0126] To address this problem, in a variation, the present method is such that when the bit depth of image cur (input bit depth) is lower than another given bit depth (given bit depth), such as the bit depth of internal samples or the bit depth of the reconstructed image, image histogram H cur It includes a step of spreading.

[0127] After increasing the bit depth, the histogram of image cur is composed of isolated "peaks" distinct from empty bins, spaced at regular intervals.

[0128] For example, the smoothing process is every kx (1< <db) 위치들에 위치된 상기 "격리된" 피크 주위에 히스토그램 피크 값들 N(k)을 분포시킴으로써 수행될 수 있으며, 이때 db = given-bit-depth(주어진-비트-깊이) - input-bit-depth(입력-비트-깊이)이다.

[0129] The distribution (spread) of the values ​​can be the same as shown on the left side of Fig. 7, or not the same as shown on the right side of Fig. 7.

[0130] The sum of the histogram values ​​within the diffusion window remains constant, equal to N(k) before and after diffusion.

[0131] In the example of Fig. 7, the dashed histogram is the image histogram after increasing the bit depth, and the full line is the diffuse image histogram. Typically, when "internal-bit-depth" = 10-bits and "input-bit-depth" = 8-bits, the diffuse window size is (1 < <db)=4이다.

[0132] In the case of unequal spread, Gaussian weights centered at "isolated" peak positions can be used to mimic the plausible / probable distribution of sample errors after coding, as shown on the right side of Fig. 7.

[0133] Various embodiments involve decoding. "Decoding" as used in this application may include all or part of the processes performed on a received encoded sequence to produce a final output suitable for display, for example.

[0134] In various embodiments, such processes typically include one or more processes performed by a decoder, e.g., entropy decoding, inverse quantization, inverse transformation, and differential decoding.

[0135] In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various embodiments described in this application, said processes including, for example, receiving information INF; deriving a combined inter-intra prediction block from an inter prediction block and an intra prediction block based on an affine transform for a current image block; and decoding an image block using the determined combined inter-intra prediction.

[0136] As additional examples, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to generally refer to a broader decoding process will be evident based on the context of the specific descriptions and is expected to be well understood by those skilled in the art.

[0137] Various implementations involve encoding.

[0138] In a manner similar to the above discussion regarding "decoding," "decoding" as used in this application may include all or part of the processes performed, for example, on an input video sequence to generate an encoded bitstream.

[0139] In various embodiments, such processes typically include one or more processes performed by an encoder, e.g., partitioning, differential encoding, transformation, quantization, and entropy encoding.

[0140] In various embodiments, such processes also or alternatively include processes performed by an encoder of the various embodiments described in this application, for example, determining an intra prediction for a current image block using a neural network applied to the context surrounding the current image block; encoding the current image block based on the intra prediction; and transmitting the encoded current image block.

[0141] As additional examples, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or to generally refer to a broader encoding process will be evident based on the context of the specific descriptions and is considered to be well understood by those skilled in the art.

[0142] When a drawing is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a drawing is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0143] Various embodiments refer to rate distortion optimization.

[0144] In particular, during the encoding process, when constraints on computational complexity are often given, a balance or trade-off between rate and distortion is usually considered. Rate-distortion optimization is typically formulated as minimizing the rate-distortion function, which is the weighted sum of rate and distortion. Different approaches exist to solve the rate-distortion optimization problem. For example, approaches include all modes or coding parameter values ​​considered, along with a complete evaluation of the associated distortion of the reconstructed signal after coding and decoding and their coding costs.

[0145] It may be based on extensive testing of encoding options. To reduce encoding complexity, faster approaches may also be used, particularly by calculating approximated distortion based on prediction or prediction residual signals rather than reconstructed ones. A combination of these two approaches may also be used, such as using approximated distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of the various techniques to perform optimization, but optimization is not necessarily a full evaluation of both coding cost and associated distortion. However, according to at least one embodiment, rate distortion optimization in encoding is eliminated because the deep intra prediction mode is the only available mode for intra prediction.

[0146] The embodiments and aspects described herein may be implemented, for example, as a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of embodiment (e.g., discussed only as a method), embodiments of the discussed features may also be implemented in other forms (e.g., a device or a program).

[0147] The device can be implemented, for example, with appropriate hardware, software, and firmware.

[0148] The methods may be implemented in, for example, a processor, generally referred to as a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device.

[0149] Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0150] References to "one embodiment," "one example," "one embodiment," or "one embodiment," as well as other variations thereof, mean that specific features, structures, properties, etc. described in relation to the embodiment are included in at least one embodiment. Accordingly, the phrases "in one embodiment," "in one example," "in one embodiment," or "in one embodiment," as well as the appearance of any other variations appearing throughout this application, do not necessarily all refer to the same embodiment.

[0151] Additionally, the present application may refer to "determining" various information. Determining information may include, for example, estimating information, calculating information, predicting information, or retrieving information from memory.

[0152] Additionally, the present application may refer to "accessing" various information. Accessing information may include, for example, receiving information, retrieving information (for example, from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0153] Additionally, the present application may refer to "receiving" various information. Receiving is intended to be a broad term, such as "accessing." Receiving information may include, for example, accessing information or retrieving information (for example, from memory). Furthermore, "receiving" is typically involved in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0154] For example, in the cases of “A / B”, “A and / or B”, and “at least one of A and B”, it will be understood that the use of any of the following “ / ”, “and / or”, and “at least one of ~” is intended to encompass the selection of only the first enumerated option (A), or the selection of only the second enumerated option (B), or the selection of both options (A and B). As an additional example, in the cases of “A, B and / or C” and “at least one of A, B, and C”, such phrases are intended to encompass the selection of only the first enumerated option (A), or the selection of only the second enumerated option (B), or the selection of only the third enumerated option (C), or the selection of only the first and second enumerated options (A and B), or the selection of only the first and third enumerated options (A and C), or the selection of only the second and third enumerated options (B and C), or the selection of all three options (A, B, and C). This can be extended to many items as listed, as is obvious to those skilled in the art of the present and related technical fields.

[0155] Additionally, as used herein, the word “signal” refers to something representing something for a corresponding decoder, in particular. For example, in some embodiments, the encoder signals a specific parameter among a plurality of parameters for deep intra prediction.

[0156] In this manner, in one embodiment, the same parameter is used on both the encoder side and the decoder side. Thus, for example, the encoder may transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already possesses not only the specific parameter but also others, signaling may be used without transmission (implicit signaling) simply to enable the decoder to know and select the specific parameter. By avoiding the transmission of any actual functions, bit savings are realized in various embodiments. It should be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder in various embodiments. The preceding relates to the verb form of the word "signal," but the word "signal" may also be used herein as a noun.

[0157] As is apparent to those skilled in the art, embodiments may generate various signals formatted to convey information that may be stored or transmitted, for example. The information may include, for example, instructions for performing a method, or data generated by one of the described embodiments. For example, a signal may be formatted to convey a bitstream of the described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information conveyed by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over various different wired or wireless links. The signal may be stored on a processor-readable medium.

[0158] A number of embodiments are described. The features of these embodiments may be provided alone or in any combination across various claim categories and types.

Claims

Claim 1 A method comprising the step of determining weighted prediction parameters for predicting image blocks, the method comprising: obtaining first weighted prediction parameters; obtaining second weighted prediction parameters from the first weighted prediction parameters based on a scaled reference image histogram derived from a reference image histogram obtained from samples of a reference image; determining a first histogram distortion between an image histogram derived from samples of an image and the reference image histogram; determining a second histogram distortion between the scaled reference image histogram and the image histogram derived by scaling the bins of the reference image histogram with the second weighted prediction parameters; and enabling the use of the second weighted prediction parameters for predicting image blocks based on the first histogram distortion and the second histogram distortion for a first component of the image. A method comprising the step of enabling the use of the weighted prediction parameters for predicting the image blocks based on the first weighted prediction parameters, the sum of absolute differences between samples of the image and samples of a scaled reference image, and the sum of absolute differences between samples of the image and samples of the reference image, for a second component of the image. Claim 2 In claim 1, the step of obtaining the second weighted prediction parameters depends on the configuration parameter. Claim 3 A method according to claim 1, further comprising the step of spreading the image histogram before obtaining the first weighted prediction parameters. Claim 4 In paragraph 3, the step of spreading the image histogram depends on the bit depth of the samples of the image and a given bit depth. Claim 5 In paragraph 3, the step of spreading the image histogram comprises the step of distributing samples of the image around the peak of the image histogram with respect to the peak. Claim 6 A method according to claim 1, wherein the first weighted prediction parameters are obtained from default weighted parameters or are obtained based on samples of the image and samples of the reference image. Claim 7 The method according to claim 1, wherein the step of obtaining the second weighted prediction parameters comprises: determining the image histogram and the reference image histogram; deriving current weighted prediction parameters from previously obtained weighted prediction parameters; deriving the scaled reference image histogram by scaling the bins of the reference image histogram with the current weighted prediction parameters; determining the histogram distortion between the image histogram and the scaled reference image histogram; and storing the current weighted prediction parameters when the histogram distortion is smaller than the histogram distortion associated with the previously obtained weighted prediction parameters. Claim 8 As an apparatus, the apparatus comprises one or more processors configured to determine weighted prediction parameters for predicting image blocks, wherein the one or more processors obtain first weighted prediction parameters; obtain second weighted prediction parameters from the first weighted prediction parameters based on a scaled reference image histogram derived from a reference image histogram obtained from samples of a reference image; determine a first histogram distortion between an image histogram derived from samples of an image and the reference image histogram; determine a second histogram distortion between the scaled reference image histogram derived by scaling bins of the reference image histogram with the second weighted prediction parameters; enable the use of the second weighted prediction parameters for predicting image blocks based on the first histogram distortion and the second histogram distortion for a first component of the image; and for a second component of the image, the first weighted prediction parameters, and the sum of absolute difference values ​​between samples of the image and samples of the scaled reference image, and An apparatus configured to enable the use of weighted prediction parameters for predicting image blocks based on the sum of absolute differences between samples of the image and samples of the reference image. Claim 9 In claim 8, the apparatus for obtaining the second weighted prediction parameters depends on the configuration parameters. Claim 10 An apparatus configured such that, in order to obtain the second weighted prediction parameters, the one or more processors determine the image histogram and the reference image histogram; derive current weighted prediction parameters from previously obtained weighted prediction parameters; derive the scaled reference image histogram by scaling the bins of the reference image histogram with the current weighted prediction parameters; determine the histogram distortion between the image histogram and the scaled reference image histogram; and store the current weighted prediction parameters when the histogram distortion is smaller than the histogram distortion associated with the previously obtained weighted prediction parameters. Claim 11 An apparatus according to claim 8, further comprising means for spreading the image histogram before obtaining the first weighted prediction parameters. Claim 12 A non-transient computer-readable storage medium comprising said instructions that, when the instructions are executed by a computer, cause said computer to perform the method of claim 1. Claim 13 delete Claim 14 delete Claim 15 delete

Images