Data processing system that modifies motion compensation information and method for decoding video data with the same

Abstract

Data processing system (100A, 100B, 100C) comprising: a first core (CORE0) configured to decode a first image (10, 610, 620, 630, 640, 650, 660) and to generate decoding history information of the decoded first image (10, 610, 620, 630, 640, 650, 660); and a second core (CORE1) configured to modify first motion compensation information (1010) contained in a second image (612, 622, 632, 642, 652, 662) based on the decoding history information of the decoded first image (10, 610, 620, 630, 640, 650, 660), and to decode the second image (612, 622, 632, 642, 652, 662) based on the decoded first image (10, 610, 620, 630, 640, 650, 660) and the modified first motion compensation information (1010), wherein the second core (CORE1) is configured to generate the decoding history information of the decoded second image (612, 622, 632, 642, 652, 662), and the first core (COREO) is configured to modify second motion compensation information (1010) contained in a third image (614, 624, 634, 644, 654, 664) based on the decoding history information of the decoded second image (612, 622, 632, 642, 652, 662), and to modify the third image (614, 624, 634, 644, 654, 664) based on the modified second motion compensation information (1010) and one of the decoded first image (610, 620, 630, 640, 650, 660) and the decoded second image (612, 622, 632, 642, 652, 662) to decode, wherein the first core (CORE0) comprises a first motion compensation information modifier (870, 920) configured to modify the second motion compensation information (1010), and the second core (CORE1) comprises a second motion compensation information modifier (870, 920) configured to modify the first motion compensation information (1010), wherein the first core (CORE0) has a first in-loop filter (790, 860A, 860B, 930) connected to the first motion compensation information modifier (870, 920), and the second core (CORE1) has a second in-loop filter (790, 860A, 860B, 930) connected to the second motion compensation information modifier (870, 920), and wherein the first in-loop filter (790, 860A, 860B, 930) is configured to perform partial in-loop filtering, which is a deblocking operation only in a block related to the modified first motion compensation information (1010), based on a first control signal received from the first motion compensation information modifier (870, 920), and the second in-loop filter (790, 860A, 860B, 930) is configured to perform partial in-loop filtering, which is a deblocking operation only in a block related to the modified second motion compensation information (1010), based on a second control signal received from the second motion compensation information modifier (870, 920).

Description

BACKGROUND 1. TECHNICAL FIELD

[0001] Example embodiments relate to a semiconductor device, and more particularly to an application processor having multiple cores for performing real-time in-loop filtering and a system having the same. 2. DESCRIPTION OF THE STATE OF THE ART

[0002] A slice and a tile in an image can reduce dependency in entropy encoding or entropy decoding. However, when a plurality of slices or tiles in the image are processed in parallel by a plurality of processors, a deblocking operation of the plurality of slices or tiles may be blocked until the processing operation of macroblocks adjacent to boundaries of the plurality of slices or tiles is completed. In addition, the plurality of processors must exchange information regarding the deblocking operation of the macroblocks adjacent to boundaries of the plurality of slices or tiles.

[0003] That is, when a plurality of divided regions in an image are processed by a plurality of processors in parallel, the deblocking operation may be blocked or delayed due to the dependencies and distributions of the plurality of divided regions in the image being processed by the plurality of processors.

[0004] WO 2016 / 060405 A1 (state of the art according to Section 3 (2) of the German Patent Act) describes a method and apparatus for parallel video decoding in a multi-core system. The goal is to improve decoding efficiency by distributing tasks across multiple cores. The focus is on optimizing hardware resource utilization and reducing latency through efficient parallelization strategies.

[0005] The scientific paper by MENG, Shengbin et al. entitled "Highly optimized implementation of HEVC decoder for general processors," published in the Proceedings of the "16th International Workshop on Multimedia Signal Processing" from September 22-24, 2014, ISBN 978-1-4799-5896-2, concerns an optimized implementation of a HEVC (High Efficiency Video Coding) decoder for general processors that utilizes SIMD (Single Instruction, Multiple Data) instructions and multithreading to maximize decoding efficiency.

[0006] The scientific paper by LIN, Jian-Liang et al., titled "Motion vector coding in the HEVC standard," published in "IEEE journal of selected topics in signal processing," Vol. 7, 2013, No. 6, ISSN 1932-4553, concerns motion vector coding techniques in the HEVC standard, focusing on methods to improve compression efficiency and reduce bitrate.

[0007] WO 2008 / 079041 A1 describes methods and apparatus for parallel video decoding and encoding, with a focus on using multiple threads to process video images simultaneously and increase efficiency. SUMMARY

[0008] According to one aspect of an exemplary embodiment, a data processing system is provided comprising a first core configured to decode a first image and to generate decoding history information of the decoded first image; and a second core configured to modify first motion compensation information contained in a second image based on the decoding history information of the decoded first image, and to decode the second image based on the decoded first image and the modified first motion compensation information.

[0009] The second core is configured to generate decoding history information of the decoded second image, and the first core is configured to modify second motion compensation information included in a third image based on the decoding history information of the decoded second image, and to decode the third image based on the modified second motion compensation information and one of the decoded first image and the decoded second image.

[0010] The first core includes a first motion compensation information modifier configured to modify the second motion compensation information, and the second core includes a second motion compensation information modifier configured to modify the first motion compensation information.

[0011] The first core has a first in-loop filter connected to the first motion compensation information modifier, and the second core has a second in-loop filter connected to the second motion compensation information modifier.

[0012] The first in-loop filter performs partial in-loop filtering, which is a deblocking operation only in a block related to the modified first motion compensation information, based on a first control signal received from the first motion compensation information modifier, and the second in-loop filter performs partial in-loop filtering, which is a deblocking operation only in a block related to the modified second motion compensation information, based on a second control signal received from the second motion compensation information modifier.

[0013] Each of the first image, the second image, and the third image may include at least one divided region.

[0014] The first image may include a first divided region, the decoding history information of the decoded first image may include decoding history information of the first divided region, and the second core may modify the first motion compensation information based on the decoding history information of the first divided region.

[0015] The modified first motion compensation information may include a modified reference coordinate corresponding to a reference coordinate included in the first motion compensation information.

[0016] The second image may include a second divided region, the decoding history information of the decoded second image may include decoding history information of the second divided region, and the first core may modify the second motion compensation information based on the decoding history information of the second divided region.

[0017] The modified second motion compensation information may include a modified reference image index modified with respect to a reference image index included in the second motion compensation information.

[0018] According to one aspect of another exemplary embodiment, a method for decoding a plurality of images in video data is provided, the method comprising receiving a first image having a first divided region; receiving a second image having a second divided region; extracting first motion compensation information included in the second image; modifying the extracted first motion compensation information; and decoding the second image based on the first divided region and the modified first motion compensation information.

[0019] Modifying the extracted first motion compensation information comprises modifying a reference coordinate of a reference image included in the first motion compensation information based on decoding history information of the reference image related to the extracted first motion compensation information.

[0020] The first image has a third divided region different from the first divided region, and the reference coordinate of the reference image is modified such that the reference coordinate is changed from a first coordinate relating to the first divided region to a second coordinate relating to the third divided region.

[0021] The method further comprises decoding a block included in the second divided region based on the first image and the modified first motion compensation information; and performing partial in-loop filtering of the decoded block.

[0022] According to one aspect of another exemplary embodiment, a method for decoding a plurality of images in video data is provided, the method comprising receiving a first image having a first divided region; receiving a second image having a second divided region; extracting first motion compensation information included in the second image; modifying the extracted first motion compensation information; and decoding the second image based on the first divided region and the modified first motion compensation information.

[0023] The method further comprises receiving a third image having a third divided region; extracting second motion compensation information included in the third image; modifying the extracted second motion compensation information; and decoding the third image based on the modified second motion compensation information.

[0024] Modifying the extracted second motion compensation information comprises modifying a reference picture index included in the second motion compensation information based on decoding history information of a reference picture referred to by the reference picture index.

[0025] The reference image index is modified such that the reference image index is changed from a second index referring to the second image to a first index referring to the first image.

[0026] The method further comprises decoding a block in the third divided region based on the second image and the modified second motion compensation information; and performing partial in-loop filtering of the decoded block.

[0027] The extracted second motion compensation information may include a reference picture index, and modifying the extracted second motion compensation information may include modifying the reference picture index and a reference coordinate of a reference picture referred to by the reference picture index included in the extracted second motion compensation information, based on decoding history information of the first divided area or the second divided area related to the reference picture index and the reference coordinate of the reference picture.

[0028] The first image may have a fourth divided region different from the first divided region, and modifying the extracted first motion compensation information may include modifying the reference image index such that the reference image index is changed from a second index relating to the second image to a first index relating to the first image; and modifying the reference coordinate of the reference image such that the reference coordinate is changed from a first coordinate relating to the fourth divided region to a second coordinate relating to the first divided region.

[0029] The method may further comprise decoding a block in the third divided region based on the first image and the modified second motion compensation information; and performing partial in-loop filtering of the decoded block.

[0030] The extracted first motion compensation information may be modified based on a number of reference images included in the extracted first motion compensation information.

[0031] The extracted first motion compensation information may be modified based on a value of a motion vector included in the extracted first motion compensation information.

[0032] The extracted first motion compensation information may be modified based on a difference in pixel values ​​at a boundary between adjacent decoded blocks before a deblocking operation.

[0033] According to one aspect of another exemplary embodiment, a data processing system is provided, comprising a first decoder configured to decode a first image and to decode first motion compensation information included in the first image, and to generate first decoding history information based on a decoding result of the decoded first image;and a second decoder configured to decode a second image and generate second motion compensation information included in the second image, and to generate second decoding history information based on a decoding result of the decoded second image; a decoding history storage device connected to the first decoder and the second decoder and configured to store the first decoding history information and the second decoding history information;and a motion compensation information modifier connected to the decoding history storage device and configured to modify the first and second motion compensation information based on the first and second decoding history and the first and second motion compensation information, wherein the second decoder is configured to decode the second image based on the modified second motion compensation information.

[0034] The first decoder may decode a third picture and generate third motion compensation information(s) included in the third picture and update the first decoding history information based on the decoding result of the third picture, and the motion compensation information modifier may modify the third motion compensation information based on the third motion compensation information and the first decoding history information.

[0035] Each of the first image, the second image, and the third image may have at least one divided region.

[0036] The motion compensation information modifier may generate the modified second motion compensation information by modifying a reference coordinate of a reference image included in the second motion compensation information.

[0037] The motion compensation information modifier may generate the modified third motion compensation information by modifying a reference image index included in the third motion compensation information.

[0038] The motion compensation information modifier may generate the modified third motion compensation information by modifying a reference image index and a reference coordinate of a reference image referenced by the reference image index included in the third motion compensation information. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other aspects will become clearer by describing exemplary embodiments in detail with reference to the accompanying drawings, in which: Fig. 1 is a block diagram of a data processing system according to an exemplary embodiment; Fig. 2 is a conceptual diagram illustrating an image having a plurality of slices; Fig. Figure 3 is a conceptual diagram illustrating a sequence of images each comprising two slices; Fig. 4 is a conceptual diagram showing a procedure for processing the sequence of images shown in Fig. 3 are illustrated by two cores; Fig. 5 is a conceptual diagram illustrating a sequence of images each comprising two groups of slices; Fig. 6 is a conceptual diagram showing a procedure for processing the sequence of images displayed in Fig. 5 is shown, illustrated by two cores; Fig. Figure 7 is a conceptual diagram illustrating a sequence of images each comprising three slices; Fig. 8 is a conceptual diagram showing a procedure or process for processing the sequence of images shown in Fig. 7 is shown by three cores; Fig. 9 is a conceptual diagram illustrating a motion vector search according to an exemplary embodiment; Fig. 10 is a conceptual diagram illustrating a sequence of images, each including two tiles, according to an example embodiment; Fig. 11 is a conceptual diagram showing a procedure for processing the sequence of images displayed in Fig. 10 is shown, illustrated by two cores; Fig. 12 is a conceptual diagram illustrating a sequence of images, each comprising two tiles; Fig. 13 is a conceptual diagram illustrating a sequence of images, each comprising four tiles respectively; Fig. 14 is a conceptual diagram showing a procedure for processing the sequence of images displayed in Fig. 13, by two cores; Fig. 15 is a block diagram showing a hardware codec of the data processing system used in Fig. 1 is shown; Fig. 16 is a block diagram of an encoder of the hardware codec used in Fig. 15 is shown; Fig. 17 is a block diagram of a decoder of the hardware codec used in Fig. 15 is shown; Fig. 18 to 22 are conceptual diagrams illustrating a method of processing video data by modifying motion compensation information according to example embodiments; Fig. 23A and Fig. 23B are conceptual diagrams illustrating partial in-loop filtering according to an example embodiment; Fig. 24 is a block diagram illustrating a decoder according to an exemplary embodiment; Fig. 25 is a block diagram illustrating a data processing system according to an exemplary embodiment; Fig. 26 is a data format illustrating an encoded bitstream including compensation information according to an exemplary embodiment; Fig. 27 is a block diagram illustrating a data processing system according to an exemplary embodiment; Fig. 28 is a block diagram illustrating a data processing system according to an exemplary embodiment; Fig. 29 is a flowchart illustrating a method for processing video data by the data processing system described in Fig. 1, Fig. 18 and Fig. 19 according to an exemplary embodiment; and Fig. 30 is a flowchart illustrating a method for processing video data according to another exemplary embodiment. DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

[0040] Various exemplary embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments are shown. The present inventive concept may, however, be embodied in various forms and should not be considered limited to the exemplary embodiments discussed herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present inventive concept to those skilled in the art. The same reference numerals may indicate the same components throughout the description. In the accompanying figures, the thicknesses of layers and regions may be exaggerated for clarity.

[0041] It will also be understood that when a layer is referred to as being "on" another layer or substrate, it may be directly on the other layer or substrate, or intervening layers may also be present. In contrast, when an element is referred to as being "directly on" another element, no intervening elements are present.

[0042] Spatially relative terms such as "beneath," "under," "lower," "above," "upper," and the like may be used herein for ease of description to describe a relationship of one element or feature to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation illustrated in the figures. For example, if the device in the figures is turned over, elements described as being "below" or "below" other elements or features would then be oriented "above" the other elements or features. Thus, the exemplary term "below" can encompass both orientations of above and below.The device may be oriented differently (rotated by 90 degrees or at other orientations) and the spatially relative descriptors used herein are interpreted accordingly.

[0043] The use of the terms "a," "an," "a," "the," and similar references in the context of describing the exemplary embodiments, particularly in the context of the following claims, should be considered to encompass both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. The terms "comprising," "having," "including," and "containing" should be considered open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present inventive concept belongs. It is understood that the use of any and all examples or exemplary terms provided herein is intended only to further clarify the present inventive concept and is not a limitation on the scope of the present inventive concept unless otherwise specified. Furthermore, unless otherwise defined, any terms defined in commonly used dictionaries should not be unduly interpreted.

[0045] The exemplary embodiments will be described with reference to perspective views, cross-sectional views, and / or plan views. Accordingly, the profile of an exemplary view may be modified according to manufacturing techniques and / or tolerances. That is, the exemplary embodiments are not intended to limit the scope of the present inventive concept, but to encompass all changes and modifications that may be caused due to a change in the manufacturing process. Accordingly, portions shown in the drawings are illustrated in schematic form, and the shapes of the portions are presented merely by way of illustration and not as a limitation.

[0046] Various exemplary embodiments of the present inventive concept will be described more fully hereinafter with reference to the accompanying drawings.

[0047] An image or a processing unit (i.e., a unit to be processed) may, in various exemplary embodiments, comprise one or more subdivided regions, for example, slices or tiles. According to exemplary embodiments, an image may be processed by one core. In other words, all subdivided regions, for example, all slices or all tiles in an image, may be processed by a single core. Here, processing may mean encoding, decoding, encoding including in-loop filtering, or decoding including in-loop filtering.

[0048] In various exemplary embodiments, an image may be divided into slices. Each slice defines a sequence of an integer number of macroblocks (MBs) or coding tree units (CTUs) for AVC or HEVC. In exemplary embodiments for HEVC, an image may also be divided into tiles. The tile defines a rectangular area of ​​an integer number of CTUs and may include CTUs contained in more than one slice.

[0049] In some example embodiments, a data processing system 100A, 100B, or 100C may be embodied in a Full High Definition (FHD) TV or Ultra High Definition (UHD) TV. Additionally, the data processing system 100A, 100B, or 100C may be compatible with a video coding standard such as H.264, High Efficiency Video Coding (HEVC), and so forth.

[0050] In some example embodiments, the data processing system 100A, 100B, or 100C may be implemented on a personal computer (PC), a smartphone, a tablet PC, a mobile internet device (MID), a laptop computer, a mobile digital media player (M-DMP), or a portable media player (PMP).

[0051] Fig. 1 is a block diagram of a data processing system according to an exemplary embodiment. Referring to Fig. 1, a data processing system 100A may include a controller 200A, a plurality of memories 301, 303, and 305, and a display 400. In some example embodiments, the data processing system 100A may further include a camera 500.

[0052] The controller 200A may be implemented as an integrated circuit (IC), a system-on-chip (SoC), a motherboard, an application processor (AP), or a mobile AP.

[0053] The controller 200A can receive and encode supplied uncompressed YUV / RGB data or decode an encoded bitstream, control operations such as read / write operations of the plurality of memories 301, 303, and 305, and transmit display data or video data to the display 400. Additionally, the controller 200A can process image data or video data output from the camera 500, and can store the processed image data or processed video data in at least one of the plurality of memories 301, 303, and 305 and / or transmit the processed image data or processed video data to the display 400.

[0054] The controller 200A may include a central processing unit (CPU) 210A, a graphics processing unit (GPU) 220, a plurality of hardware codecs 230-1 through 230-N, where N is a natural number equal to or greater than two, a memory interface (I / F) 240, a display interface (I / F) 260, and a receiver interface (I / F) 270. If the data processing system 100A further includes a camera 500, the controller 200A may further include a camera interface (I / F) 250.

[0055] The CPU 210A, the GPU 220, the plurality of hardware codecs 230-1 through 230-N, the memory interface 240, the display interface 260, and the receiver interface 270 may transmit or receive data from / to each other via a bus 201. The camera interface 250 may be connected to and transmit / receive data from / to the other components, for example, 210A and / or 220.

[0056] The CPU 210A may have one or more cores. The CPU 210A may control the overall operation of the controller 200A. For example, the CPU 210A may execute program code or applications to control the operation of the GPU 220, the plurality of hardware codecs 230-1 through 230-N, the memory interface 240, the camera interface 250, the display interface 260, and / or the receiver interface 270.

[0057] The GPU 220 can process 2D or 3D graphics data. The graphics data processed by the GPU 220 can be stored in at least one of the plurality of memories 301, 303, and 305 or can be displayed by the display 400.

[0058] The plurality of hardware codecs 230-1 to 230-N may be substantially the same in terms of structure and operation. In some example embodiments, each of the plurality of hardware codecs 230-1 to 230-N may be referred to as a core. Each of the plurality of hardware codecs 230-1 to 230-N may further include one of memories MEM1 to MEMN, respectively. Each of memories MEM1 to MEMN may store information from one or more encoded and / or decoded images and / or perform in-loop filtering of the one or more encoded and / or decoded images.

[0059] The memory interface 240 may include a first memory interface 241, a second memory interface 243, and a third memory interface 245. For example, if the first memory 301 of the plurality of memories 301, 303, 305 is a dynamic random access memory (DRAM), the first memory interface 241 may be a DRAM interface. If the second memory 303 is a NAND flash memory, the second memory interface 243 may be a NAND flash interface. If the third memory 305 is an embedded multimedia card (eMMC), the third memory interface may be an eMMC interface. For ease of description in Fig. 1 illustrates three memory interfaces 241, 243, and 245 and the three memories 301, 303, and 305; however, this is only an example, and the number of memories is not particularly limited. Accordingly, a technical concept is not limited or restricted by the number of memory interfaces and memories connected to the memory interfaces.

[0060] The display interface 260 may transmit display data, such as video data, to the display 400 under control of the CPU 210A, the GPU 220, or any of the plurality of hardware codecs 230-1 through 230-N. In some exemplary embodiments, the display interface 260 may be implemented as a serial display interface (DSI) of a mobile industrial processor interface MIPI. ®, an embedded display port (eDP) interface, or a high-definition multimedia interface (HDMI), etc. However, these interfaces are only examples, and the display interface 260 is not limited thereto.

[0061] Receiver interface 270 may receive uncompressed image data (RGB / YUV) and / or an encoded bitstream over a wired communication network or a wireless communication network. In some example embodiments, receiver interface 270 may be implemented as a Universal Serial Bus (USB) interface, an Ethernet interface, a Bluetooth interface, a Wi-Fi interface, or a Long Term Evolution (LTE™) interface, etc. However, these interfaces are only examples, and receiver interface 270 is not limited thereto.

[0062] The camera interface 250 may receive image data output from the camera 500. In some exemplary embodiments, the camera interface 250 may be a serial camera interface (CSI) of a MIPI ® However, this interface is only exemplary, and the camera interface 250 is not limited thereto. The camera 500 may be a camera module having a CMOS image sensor.

[0063] Fig. 2 is a conceptual diagram illustrating an image having a plurality of slices.

[0064] Referring to Fig. 2, an image 10 may comprise a plurality of slices SLICE1 to SLICE3, and each of the plurality of slices SLICE1 to SLICE3 may comprise a plurality of blocks. For example, as in the example of Fig. 2, SLICE1 may include blocks 0-21, SLICE2 may include blocks 22-39, and SLICE3 may include blocks 40-63. A number of blocks included in each of the plurality of slices SLICE1 to SLICE3 may be different from each other. Alternatively, the number of blocks included in each of the plurality of slices SLICE1 to SLICE3 may be the same for each slice. The image 10 may be a frame to be encoded, an original image, a raw bitstream of an image to be encoded, or a frame to be decoded, an encoded image or an encoded bitstream EB, etc. Fig. Figure 2 shows an image that has 64 blocks, i.e., blocks 0-63. However, this is only an example, and the number of blocks may be more or less than 64.

[0065] In the H.264 video coding standard, the block may be a macroblock. In a high-efficiency video coding (HEVC) standard, the block may be a coding tree unit (CTU). In some example embodiments, the image 10 may be a frame to be encoded, an original image, a raw bitstream of an image to be encoded or a frame to be decoded, an encoded image, an encoded bitstream EB, etc.

[0066] An encoder can encode the blocks in a picture and a decoder can decode the blocks in a picture sequentially from a block 0 to a block 63.

[0067] After a first block 0 is processed in the first slice SLICE1, the encoder or decoder does not perform in-loop filtering of the edge of the processed first block 0 because there is no block before the processed first block 0. On the other hand, after processing a second block 1 in the first slice SLICE1, the encoder or decoder can perform in-loop filtering of the edge of the first block 0 and the second block 1 based on first video data of the processed first block 0 and second video data of the processed second block 1. This means that the encoder or decoder can perform in-loop filtering at a block boundary. Here, a block boundary refers to a boundary between two blocks.

[0068] After a 23rd encoded block 22 is processed in the second slice SLICE2, the encoder or decoder can perform in-loop filtering of the edge of the 23rd processed block 22 in the second slice SLICE2 and preceding blocks 14 and 21 in the first slice SLICE1 based on video data of blocks 14, 21, and 23. This means that the encoder or decoder can perform in-loop filtering on a write boundary.

[0069] After a 41st block 40 is processed in the third slice SLICE3, the encoder or decoder may perform in-loop filtering of the edge of the 41st processed block 40 in the third slice SLICE3 and preceding blocks 32 in the first slice SLICE2 based on video data of the 33rd blocks 32. In other words, in-loop filtering of a current processed block may be performed using at least one of an upper processed block and a left processed block adjacent to the current block.

[0070] According to exemplary embodiments, a decoder having an in-loop filter that meets a video coding standard of H.264 may perform in-loop filtering for each block. According to exemplary embodiments of the present inventive concept, a decoder having an in-loop filter that meets a video coding standard of HEVC may perform in-loop filtering for each block. For example, a decoder may perform in-loop filtering of a plurality of blocks adjacent to a currently decoded block at a block boundary or perform in-loop filtering of all pixels in the currently decoded block according to a video coding standard of Sample Adaptive Offset (SAO).

[0071] The in-loop filtering according to the H.264 video coding standard may be performed based on a motion vector, prediction mode information, and / or a quantization parameter, etc. The in-loop filtering according to the HEVC video coding standard may be performed based on a motion vector, prediction mode information, a quantization parameter, and / or a plurality of parameters of an SAO filter, etc. The prediction mode may include intra-prediction mode information or inter-prediction mode information.

[0072] Fig. Figure 3 is a conceptual diagram illustrating a sequence of images, each comprising two slices. Fig. 4 is a conceptual diagram showing a procedure or process for processing the sequence of images displayed in Fig. 3 is illustrated by two cores.

[0073] With reference to the Fig. 1, Fig. 3 and Fig. 4, the controller 200A may include two hardware codecs 230-1 and 230-2. A first core CORE1 may be a first hardware codec 230-1 and a second core CORE1 may be a second hardware codec 230-2. The encoder or decoder described with reference to Fig. 2 may be included in the first core COREO and the second core CORE1.

[0074] Referring to Fig. 3, each of the plurality of images 610, 612, 614, ..., and 616 may each include two slices 610-1 ~ 610-2, 612-1 ~ 612-2, 614-1 ~ 614-2, ..., and 616-1 ~ 616-2. Each of the plurality of images 610, 612, 614, ..., and 616 may each include a first processing unit 610-1, 612-1, 614-1, ..., and 616-1, and a second processing unit 610-2, 612-2, 614-2, ..., and 616-2. As used in the present disclosure, "processing unit" refers to a unit of data to be processed. For example, each of the plurality of images 610, 612, 614, ... and 616 may mean a frame to be encoded, an original image, a raw bitstream of an image to be encoded or a frame to be decoded, an encoded image or encoded bitstream EB received by the receiver interface 270, etc.

[0075] In some example embodiments, the CPU 210A may control the receiver interface 270 such that the odd image may be assigned to the first core CORE1 and an even image may be assigned to the second core CORE1.

[0076] In some example embodiments, the CPU 210A may control the first core COREO and the second core CORE1 such that odd images are assigned to the first core COREO and even images are assigned to the second core CORE1.

[0077] Referring to Fig. 4, in a first time period T1, the first core COREO may process the first slice 610-1 of the first image 610. The first slice 612-1 of the second image 612 may be processed by the second core CORE1 in a second time period T2 after the processing of the first slice 610-1 of the first image 610 has been completed by the first core COREO in the first time period T1.

[0078] In the second time period T2, the first core COREO may process the second slice 610-2 of the first image 610, and the second core CORE1 may process the first slice 612-1 of the second image 612. The processing of the first slice 612-1 and the second slice 610-2 may be performed in parallel in the second time period T2. When used in this description, "processing... performed in parallel" means that there is an overlap in the processing. Accordingly, here "parallel" means that there is an overlap in the processing of the first slice 612-1 and the second slice 610-2. That is, while dotted lines in the Fig. 4, Fig. 6, Fig. 8, Fig. 11, Fig. 14, these lines are provided for convenience only and are not intended to suggest that the start or end times for processing coincide exactly. For example, the dotted lines in Fig. 4 is not intended to suggest that the start times or end times for processing the first slice 612-1 and the second slice 610-2 exactly coincide. However, in some example embodiments, the start times may exactly coincide and / or the end times may exactly coincide.

[0079] Additionally, in the second time period T2, the first core COREO may perform in-loop filtering at a boundary of the first slice 610-1 and the second slice 610-2 of the first image 610 for each block based on a processing result of the first slice 610-1 in the first image 610. As described with reference to Fig. 2, the first core COREO may perform in-loop filtering of a currently processed block using at least one of an upper processed block and a left processed block adjacent to the current block. That is, the first core COREO may perform the processing of the second slice 610-2 and the in-loop filtering of the first slice 610-1 and the second slice 610-2 in the second time period T2, so that no additional time is required for the in-loop filtering.

[0080] In a third time period T3, the first core CORE1 may process the first slice 614-1 of the third image 614, and the second core CORE1 may process the second slice 612-2 of the second image 612. The processing of the first slice 614-1 of the third image 614 and the second slice 612-2 of the second image 612 may be performed in parallel in the second time period T3.

[0081] Additionally, in the third time period T3, the second core CORE1 may perform in-loop filtering at a boundary of the first slice 612-1 and the second slice 612-2 for each block based on a processing result of the first slice 612-1 in the second image 612. That is, the second core CORE1 may perform the processing of the second slice 612 and the in-loop filtering of the edge of the first slice 612-1 and the second slice 612-2 of the second image 612 in the third time period T3, so that no additional time is required for the in-loop filtering. The second core CORE1 may therefore have a one-slice delay, as shown in Fig. 4. That is, the second core CORE1 may begin processing the first slice 612-1 of the second image 612 after the first core COREO has finished processing the first slice 610-1 of the first image 610.

[0082] Fig. Figure 5 is a conceptual diagram illustrating a sequence of images each having two groups of discs. Fig. Figure 6 is a conceptual diagram illustrating a procedure or process for changing the sequence of images displayed in Fig. 5 is shown, to be processed by two cores.

[0083] The 200A controller, which is Fig. 1, may include two hardware codecs 230-1 and 230-2, each corresponding to a first core COREO and a second core CORE1. The sequence of images may, for example, include images 620, 622, 624, and 626.

[0084] The first core COREO may divide the image 620 into a first processing unit and a second processing unit based on a processing complexity or size of a plurality of divided regions in the image 620. In the Fig. 5 and Fig. 6, the first processing unit may comprise two slices SLICE1-1 and SLICE1-2, and the second processing unit may comprise three slices SLICE1-3 to SLICE1-5. The first processing unit is shown above the thick line in Fig. 5, and the second processing unit is shown under the thick line in Fig. 5 shown. Fig. 5 shows that each of the first processing units has two slices. For example, the first image 620 has a first processing unit with SLICE1-1 and SLICE1-2. However, this is only an example, and the first processing unit may have a number of slices that is fewer or greater than two slices, depending on the complexity or size of the plurality of divided regions.

[0085] The second core CORE1 can divide the image 622 into a first processing unit and a second processing unit based on the processing complexity of the image 622. In the Fig. 5 and Fig. 6, the first processing unit may comprise two slices SLICE2-1 and SLICE2-2, and the second processing unit may comprise three slices SLICE2-3 to SLICE2-5.

[0086] For example, when images 620, 622, 624, and 626 are not divided into a plurality of slices or tiles, or when a plurality of divided regions (e.g., slices or tiles) have different processing complexities, it is advantageous to define a plurality of divided regions for parallel processing by the plurality of cores COREO and CORE1 based on the processing complexities. Here, the first core COREO and the second core CORE1 can divide the plurality of divided regions of images 620, 622, 624, and 626 and process them in parallel. The processing complexity can be defined by a decoding time and / or an encoding time and / or a size of the slices in units of blocks of an image, etc.

[0087] If m is a natural number equal to or greater than four, then Fig. 5, the first core COREO divides the image 624 into a first processing unit and a second processing unit based on the processing complexity of the image 624. In the Fig. 5 and Fig. 6, the first processing unit may comprise two slices SLICE3-1 and SLICE3-2, and the second processing unit may comprise three slices SLICE1-3 to SLICE1-5.

[0088] The second core CORE1 can divide the image 626 into a first processing unit and a second processing unit based on the processing complexity of the image 626. In the Fig. 5 and Fig. 6, the first processing unit may comprise two slices SLICEm-1 and SLICEm-2, and the second processing unit may comprise three slices SLICEm-3 to SLICEm-5.

[0089] Although each image 620, 622, 624, ... or 626 is illustrated to be divided into a first processing unit and a second processing unit for the two cores COREO and CORE1 respectively for the purpose of expediently describing the Fig. 5 and Fig. 6, a number of processing units in an image may vary according to a number of cores for processing the units.

[0090] For example, a processing time for processing the three slices SLICE1-3 through SLICE1-5 by the first core CORE1 and a processing time for processing the two slices SLICE2-1 through SLICE2-2 by the second core CORE1 may be substantially the same. Here, the meaning of "substantially the same" may include "the same, nearly the same, almost the same, or the same with a permissible margin of difference."

[0091] As in Fig. 5, each image 620, 622, 624, ... 626 may comprise five slices SLICE1-1 to SLICE1-5, SLICE2-1 to SLICE2-5, SLICE3-1 to SLICE3-5, ... SLICEm-1 to SLICEm-5, respectively. For example, each image 620, 622, 624, ... 626 may be a frame to be coded, an original image, a raw bitstream of an image to be coded or a frame to be decoded, an encoded image, or an encoded bitstream EB received via the receiver interface 270, etc.

[0092] Now Fig. 6, in a first time period T11, the first core COREO may process the first slice SLICE1-1 and the second slice SLICE1-2 of the first image 620. In the first time period T11, the first core COREO may perform in-loop filtering at a boundary of the slices SLICE1-1 and SLICE1-2 for each block based on a processing result of the slice SLICE1-1.

[0093] In a second time period T12, the first core CORE1 may process the third slice SLICE1-3 to the fifth slice SLICE1-5 of the first image 620, and the second core CORE1 may process the first slice SLICE2-1 and the second slice SLICE2-2 of the second image 622. The processing of the third slice SLICE1-3 to the fifth slice SLICE1-5 and the processing of the first slice SLICE2-1 and the second slice SLICE2-2 may be performed in parallel in the second time period T12.

[0094] In the second time period T12, the first core COREO may perform in-loop filtering at a boundary of the slices SLICE1-2 and SLICE1-3 for each block based on the processing result of the slices SLICE1-2. In the second time period T12, the first core COREO may perform in-loop filtering at a boundary of the slices SLICE1-3 and SLICE1-4 for each block based on the processing result of the slice SLICE1-3, and may perform in-loop filtering at a boundary of the slices SLICE1-4 and SLICE1-5 for each block based on the processing result of the slice SLICE1-4. Additionally, in the second time period T12, the second core CORE1 may perform in-loop filtering at a boundary of the slices SLICE2-1 and SLICE2-2 for each block based on the processing result of the slice SLICE2-1.

[0095] This means that in the second time period T12, the first core COREO can perform in-loop filtering for the three slices SLICE1-3 to SLICE1-5, and the second core CORE1 can perform in-loop filtering for the two slices SLICE2-1 and SLICE2-2, so that no additional time is required for in-loop filtering, for example, no additional time for collecting distributed slices across the first core COREO and the second core CORE1.

[0096] In a third time period T13, the first core COREO may process the first slice SLICE3-1 and the second slice SLICE3-2 of the third image 624. In the third time period T31, the first core COREO may perform in-loop filtering at a boundary of the slices SLICE3-1 and SLICE3-2 for each block based on the processing result of the slice SLICE3-1.

[0097] In the third time period T13, the first core CORE1 may process the first slice SLICE3-1 and the second slice SLICE3-2 of the third image 624, and the second core CORE1 may process the third slice SLICE2-3 through the fifth slice SLICE2-5 of the second image 622. The processing of the third slice SLICE2-3 through the fifth slice SLICE2-3 and the processing of the first slice SLICE3-1 and the second slice SLICE3-2 may be performed in parallel in the third time period T13.

[0098] In the third time period T13, the first core CORE1 may perform in-loop filtering at a boundary of the slices SLICE3-1 and SLICE3-2 for each block based on the processing result of the slices SLICE3-1. Additionally, in the third time period T13, the second core CORE1 may perform in-loop filtering at a boundary of the slices SLICE2-2 and SLICE2-3 for each block based on the processing result of the slice SLICE2-2. In the third time period T13, the second core CORE1 may perform in-loop filtering at a boundary of the slices SLICE2-3 and SLICE2-4 for each block based on the processing result of the slice SLICE2-3 and may perform in-loop filtering at a boundary of the slices SLICE2-4 and SLICE2-5 for each block based on the processing result of the slice SLICE2-4.

[0099] This means that in the third time period T13, the first core COREO can perform in-loop filtering for the two slices SLICE3-1 and SLICE3-2, and the second core CORE1 can perform in-loop filtering for the three slices SLICE2-3 to SLICE2-5, so that no additional time is used for in-loop filtering, for example, no additional time for collecting distributed slices across the first core COREO and the second core CORE1.

[0100] Fig. Figure 7 is a conceptual diagram illustrating a sequence of images, each containing three slices. Fig. 8 is a conceptual diagram showing a procedure or process for processing the sequence of images displayed in Fig. 5 is illustrated by three cores.

[0101] The 200A controller, which is Fig. 1, may comprise three hardware codecs 230-1, 230-2, and 230-3, which may each correspond to a first core CORE1, a second core CORE1, and a third core CORE2. The encoder and / or decoder shown in Fig. 2 can be embedded in each of the cores COREO, CORE1 and CORE2.

[0102] As in Fig. As shown in Figure 7, each image 630, 632, 634, 636, and 638 may each include three slices. Each image 630, 632, 634, 636, and 638 may include a first processing unit, a second processing unit, and a third processing unit. That is, each of the first processing unit, the second processing unit, and the third processing unit may each correspond to one of the three slices. Each image 630, 632, 634, 636, and 638 may be a frame to be coded, an original image, a raw bitstream of an image to be coded, a frame to be decoded, an encoded image or an encoded bitstream EB received by the receiver interface 270, etc.

[0103] Itself Fig. 8, in a first time period T21, the first core COREO can process the first slice SLICE1-1 of the first image 630.

[0104] In a second time period T22, the first core CORE1 may process the second slice SLICE1-2 of the first image 630, and the second core CORE1 may process the first slice SLICE2-1 of the second image 632. The processing of the second slice SLICE1-2 and the processing of the first slice SLICE2-1 may be performed in parallel in the second time period T22.

[0105] In the second time period T22, the first core COREO may perform in-loop filtering at a boundary of the slices SLICE1-1 and SLICE1-2 for each block based on the processing result of the slice SLICE1-1.

[0106] For parallel processing, a one (1) slice delay can be assigned to the second core CORE1 and a two (2) slice delay can be assigned to the third core CORE2.

[0107] In a third time period T23, the first core CORE1 may process the third slice SLICE1-3 of the first image 630, the second core CORE1 may process the second slice SLICE2-2 of the second image 632, and the third core CORE2 may process the first slice SLICE3-1 of the third image 634.

[0108] In the third time period T23, the processing of the third slice SLICE1-3, the processing of the second slice SLICE2-2 and the first slice SLICE1-3 can be carried out in parallel in the third time period T23.

[0109] In the third time period T23, the first core COREO may perform in-loop filtering at a boundary of the slices SLICE1-2 and SLICE1-3 for each block based on the processing result of the slice SLICE1-2, and the second core CORE1 may perform in-loop filtering at a boundary of the slices SLICE2-1 and SLICE2-2 for each block based on the processing result of the slice SLICE2-1.

[0110] In a fourth time period T24, the first core COREO may process a first slice SLICE4-1 of the fourth image 636, the second core CORE1 may process the third slice SLICE2-3 of the second image 632, and the third core CORE2 may process the second slice SLICE3-2 of the third image 634. This means that once the first core COREO completes processing the slices of the first image 630, the first core COREO advances to the next image in the sequence of images, in this case, the fourth image 636.

[0111] In the fourth time period T24, the processing of the first slice SLICE4-1, the processing of the third slice SLICE2-3 and the processing of the second slice SLICE3-2 can be carried out in parallel.

[0112] In the fourth time period T24, the second core CORE1 may perform in-loop filtering at a boundary of the slices SLICE2-2 and SLICE2-3 for each block based on the processing result of the slice SLICE2-2, and the third core CORE2 may perform in-loop filtering at a boundary of the slices SLICE3-1 and SLICE3-2 for each block based on the processing result of the slice SLICE3-1.

[0113] In a fifth time period T25, an operation of each of the cores COREO, CORE1 and CORE2 may continue to process respective slices in a similar way as in the operation in the first to fourth time periods T21 to T24.

[0114] Fig. Figure 9 is a conceptual diagram illustrating a motion vector search. Referring to the Fig. 3, Fig. 4 and Fig. 9, if a current processing slice is a first slice 614-1 of a third image 614, the first core COREO may search processed slices 610-1, 610-2, and 612-1 to find the best motion vector. For example, since each of slices 612-2 and 614-1 is processed in parallel by the different cores COREO and CORE1, the first core may not refer to the second slice 612-2 of the second image 612 now being processed. Likewise, slice 614-1 may not be able to refer to a few bottom pixels of slice 612-1 because the boundary between slices 612-1 and 612-2 may be under an in-loop filtering operation at the time slice 614-1 is processed (in the H.264 standard, the few bottom pixels may be six pixels).

[0115] The data processing system 100A may encode the third image 614 to avoid or reduce the dependency among the slices SLICE2-2 and SLICE3-1 in parallel processing by determining a reference image and a coordinate of a matching block (or a reference block) based on the dependency. For example, when encoding the first slice 614-1, a motion vector of a macroblock in the first slice 614-1 may be determined to avoid dependency by causing the macroblock to refer to one of the slices 610-1, 610-2, and 612-1 instead of the second slice 612-2, with some restrictions on the ground pixels processed by the in-loop filtering operation. As a result, the dependency in parallel processing can be reduced.

[0116] Data processing system 100A may generate and reference a motion restriction slice group to reduce dependency between slices processed in parallel when encoding the slices. The motion restriction slice group may include information to restrict a reference image or a reference region of a previous image referenced by a current processing slice. The motion restriction slice group may be stored in a memory 301, 303, or 305 in data processing system 100A or in a memory in each of cores 230-1 through 230-N.

[0117] Fig. 10 is a conceptual diagram illustrating a sequence of images, each including two tiles, according to an example embodiment. Fig. 11 is a conceptual diagram showing a procedure for processing the sequence of images displayed in Fig. 10 is illustrated by two cores.

[0118] An arrow shown in each of the plurality of images 640, 642, 644, and 646 indicates a processing direction or a processing order of a plurality of tiles in the plurality of images 640, 642, 644, and 646.

[0119] The 200A controller in Fig. 1 may include two hardware codecs 230-1 and 230-2. A first core CORE0 may be a first hardware codec 230-1, and a second core CORE1 may be a second hardware codec 230-2. In some example embodiments, all tiles in an image may be processed by a single core.

[0120] Referring to Fig. 10, each of the plurality of images 640, 642, 644, 646 may include two tiles TILE1-1 and TILE1-2, TILE2-1 and TILE2-2, TILE3-1 and TILE3-2, and TILE4-1 and TILE4-2, respectively. Each of the tiles TILE1-1 and TILE1-2, TILE2-1 and TILE2-2, TILE3-1 and TILE3-2, and TILE4-1 and TILE4-2 may include one or more blocks. Each of the plurality of images 640, 642, 644, 646 may include a first processing unit TILE1-1, TILE2-1, TILE3-1, and TILE4-1, and a second processing unit TILE2-1, TILE2-2, TILE3-2, and TILE4-2. That is, for example, a first processing unit of image 640 includes TILE1-1, and a second processing unit of image 640 includes TILE1-2.Here, each of the plurality of images 640, 642, 644, 646 may be a frame to be coded, an original image, a raw bitstream of an image to be coded, a frame to be decoded, an encoded image or an encoded bitstream EB received by the receiver interface 270, etc.

[0121] Operations of the first core COREO and the second core CORE1 are performed with reference to the Fig. 1, Fig. 2, Fig. 10 and Fig. 11 are described.

[0122] In a first time period T31, the first core CORE1 can process the first tile TILE1-1 of the first image 640. For parallel processing, a tile delay can be assigned to the second core CORE1.

[0123] In a second time period T32, the first core CORE1 may process the second tile TILE1-2 of the first image 640, and the second core CORE1 may process the first tile TILE2-1 of the second image 642. The processing of the first tile TILE2-1 and the second tile TILE1-2 may be performed in parallel in the second time period T2. As in the case discussed above, "parallel" here means that there is an overlap in the processing of the first tile TILE2-1 and the second tile TILE1-2. This means that, while dotted lines in Fig. 4, Fig. 6, Fig. 8, Fig. 11, Fig. 14, these lines are provided only for convenience of description and are not intended to suggest that the start times or end times for a processing operation coincide exactly. For example, the dotted lines in Fig. 5 is not intended to suggest that the start times or end times for processing the first tile TILE2-1 and the second tile TILE1-2 exactly coincide. However, in some example embodiments, the start times may exactly coincide and / or the end times may exactly coincide.

[0124] Additionally, in the second time period T32, the first core COREO may perform in-loop filtering at a boundary of the tiles TILE1-1 and TILE1-2 for each block based on a processing result of the first tile TILE1-1 in the first image 640. That is, in the second time period T32, the first core COREO may perform processing of the second tile TILE1-2 and in-loop filtering of the tiles TILE1-1 and TILE1-2, so no additional time is used for in-loop filtering.

[0125] In a third time period T33, the first core CORE1 can process the first tile TILE3-1 of the third image 644, and the second core CORE1 can process the second tile TILE2-2 of the second image 642. The processing of the first tile TILE3-1 and the second slice TILE2-2 can be performed in parallel in the third time period T33.

[0126] Additionally, in the third time period T33, the second core CORE1 may perform in-loop filtering at a boundary of the tiles TILE2-1 and TILE2-2 for each block based on a processing result of the first tile TILE2-1 in the second image 642. That is, in the third time period T33, the second core CORE1 may perform processing of the second tile TILE2-2 and in-loop filtering of the tiles TILE2-1 and TILE2-2, so no additional time is used for in-loop filtering.

[0127] In a fourth time period T34, each core of the cores COREO and CORE1 may continue to process respective tiles in a similar way as in the operation in the third time period T33.

[0128] Fig. 12 is a conceptual diagram illustrating a sequence of images, each having two tiles, each partitioned using horizontal partitioning.

[0129] An arrow shown in each of a plurality of images 650, 652, 654, and 656 illustrates a processing direction or processing order of a plurality of tiles in the plurality of images 650, 652, 654, and 656.

[0130] The 200A controller in Fig. 1 may include two hardware codecs 230-1 and 230-2. A first core COREO may be a first hardware codec 230-1, and the second core CORE1 may be a second hardware codec 230-2. Operations of the first core COREO and the second core CORE1 are described with reference to Fig. 1, Fig. 2, Fig. 11 and Fig. 12 are described.

[0131] Referring to Fig. 12, each of the plurality of images 650, 652, 654, 656 may include two tiles, TILE1-1 and TILE1-2, TILE2-1 and TILE2-2, TILE3-1 and TILE3-2, and TILE4-1 and TILE4-2, respectively. Each of the tiles TILE1-1 and TILE1-2, TILE2-1 and TILE2-2, TILE3-1 and TILE3-2, and TILE4-1 and TILE4-2 may include one or more blocks. Each of the plurality of images 650, 652, 654, 656 may each have a first processing unit TILE1-1, TILE2-1, TILE3-1, and TILE4-1, and a second processing unit TILE2-1, TILE2-2, TILE3-2, and TILE4-2. Here, each of the plurality of images 650, 652, 654, 656 may be a frame to be coded, an original image, a raw bitstream of an image to be coded, a frame to be decoded, an encoded image or an encoded bitstream EB received by the receiver interface 270, etc.

[0132] In a first time period T31, the first core CORE1 may process the first tile TILE1-1 of the first image 650. For parallel processing, a tile delay may be assigned to the second core CORE1.

[0133] In a second time period T32, the first core CORE1 can process the second tile TILE1-2 of the first image 650, and the second core CORE1 can process the first tile TILE2-1 of the second image 652. The processing of the first tile TILE2-1 and the second tile TILE1-2 can be performed in parallel in the second time period T32.

[0134] Additionally, in the second time period T32, the first core COREO may perform in-loop filtering at a boundary of the tiles TILE1-1 and TILE1-2 for each block based on a processing result of the first tile TILE1-1 in the first image 650. That is, in the second time period T32, the first core COREO may perform processing of the second tile TILE1-2 and in-loop filtering of the tiles TILE1-1 and TILE1-2, so no additional time is used for in-loop filtering.

[0135] In a third time period T33, the first core CORE1 may process the first tile TILE3-1 of the third image 654, and the second core CORE1 may process the second tile TILE2-2 of the second image 652. The processing of the first tile TILE3-1 and the second slice TILE2-2 may be performed in parallel in the third time period T33.

[0136] Additionally, in the third time period T33, the second core CORE1 may perform in-loop filtering at a boundary of the tiles TILE2-1 and TILE2-2 for each block based on a processing result of the first tile TILE2-1 in the second image 652. That is, in the third time period T33, the second core CORE1 may perform processing of the second tile TILE2-2 and in-loop filtering of the tiles TILE2-1 and TILE2-2, so no additional time is used for in-loop filtering.

[0137] In a fourth time period T34, each core of the cores COREO and CORE1 may continue to process respective tiles in a similar way as in the operation in the third time period T33.

[0138] Fig. Figure 13 is a conceptual diagram illustrating a sequence of images, each of which comprises four tiles having both horizontal and vertical partitions. Fig. 14a is a conceptual diagram showing a procedure or process for processing the sequence of images displayed in Fig. 13 are illustrated by two cores.

[0139] An arrow shown in each of a plurality of images 660, 662, and 664 illustrates a processing direction or processing order of a plurality of tiles in the plurality of images 660, 662, and 664.

[0140] The 200A controller in Fig. 1 may include two hardware codecs 230-1 and 230-2. A first core COREO may be a first hardware codec 230-1, and a second core CORE1 may be a second hardware codec 230-2. In some exemplary embodiments, all tiles in an image may be processed by a single core. Operations of the first core COREO and the second core CORE1 are described with reference to Fig. 1, Fig. 2, Fig. 13 and Fig. 14 are described.

[0141] Referring to Fig. 13, each of the plurality of images 660, 662, and 664 may include four tiles TILE1-1 through TILE1-4, TILE2-1 through TILE2-4, and TILE3-1 through TILE3-4, respectively. Each of the tiles TILE1-1 through TILE1-4, TILE2-1 through TILE2-4, and TILE3-1 through TILE3-4 may include one or more blocks. Each of the plurality of images 660, 662, and 664 may include a first processing unit having two tiles TILE1-1 and TILE1-2, TILE2-1 and TILE2-2, and TILE3-1 and TILE3-2, respectively, and a second processing unit having two tiles TILE1-3 and TILE1-4, TILE2-3 and TILE2-4, and TILE3-3 and TILE3-4, respectively. Here, each of the plurality of images 660, 662, and 664 may be a frame to be coded, an original image, a raw bitstream of an image to be coded, a frame to be decoded, an encoded image, or an encoded bitstream EB received by the receiver interface 270, etc.

[0142] In a first time period T51, the first core CORE1 may process the first tile TILE1-1 and the second tile TILE1-2 of the first image 660 and perform in-loop filtering at a boundary of the tiles TILE1-1 and TILE1-2 of the first image 660 for each block based on a processing result of the first tile TILE1-1. For parallel processing, two tile delays may be assigned to the second core CORE1.

[0143] In a second time period T52, the first core CORE1 may process the third tile TILE1-3 and fourth tile TILE1-4 of the first image 660, and the second core CORE1 may process the first tile TILE2-1 and the second tile TILE2-2 of the second image 662. The processing of the third tile TILE1-3 and the fourth tile TILE1-4 and the processing of the first tile TILE2-1 and the second tile TILE2-2 may be performed in parallel in the second time period T52. As in the case discussed above, “parallel” here means that there is an overlap in the processing of the tiles. That is, while dotted lines in the Fig. 4, Fig. 6, Fig. 8, Fig. 11, Fig. 14, these lines are provided for convenience of description only and are not intended to suggest that the start times or end times for processing exactly coincide. However, in some example embodiments, the start times may exactly coincide and / or the end times may exactly coincide.

[0144] In addition, in the second time period T52, the first core COREO may perform in-loop filtering at a boundary of the tiles TILE1-1 and TILE1-3 for each block based on a processing result of the first tile TILE1-1 in the first image 660.

[0145] In the second time period T52, the first core COREO may perform in-loop filtering at a boundary of the tiles TILE1-2 and TILE1-4 for each block based on a processing result of the first tile TILE1-2 in the first image 660.

[0146] In the second time period T52, the first core COREO may perform in-loop filtering at a boundary of the tiles TILE1-3 and TILE1-4 for each block based on a processing result of the first tile TILE1-3 in the first image 660.

[0147] In the second time period T52, the second core CORE1 may perform in-loop filtering at a boundary of the tiles TILE2-1 and TILE2-2 of the second image 662 for each block based on a processing result of the first tile TILE2-1.

[0148] This means that the in-loop filtering is performed on the boundaries of the processed tiles in the second time period T52, so that no additional time is used for in-loop filtering.

[0149] In a third time period T53, the cores COREO and CORE1 may continue to process respective tiles in a similar way as in the operation in the second time period T52.

[0150] Fig. 15 is a block diagram showing a hardware codec used in Fig. 1 is shown.

[0151] With reference to the Fig. 1 and Fig. 15, the hardware codec 230-1 may include a selection circuit 701, an encoder 700, a decoder 800, and a memory MEM1.

[0152] The selection circuit 701 can output image data or video data INV to the encoder 700 in response to a selection signal SEL when the selection signal SEL has a first level, for example, a low level. The selection circuit 701 can output an encoded bit stream EB to the decoder 800 in response to a selection signal SEL when the selection signal has a second level, for example, a high level.

[0153] The selection signal SEL may be provided by the CPU 210A. The image data or the video data INV may be provided by the camera interface 250, and the encoded bit stream EB may be provided by the receiver interface 270 or the memory interface 240.

[0154] The encoder 700 may encode the output image data or output video data INV from the selecting unit 701 and output the encoded image data or the encoded video data to the decoder 800 through an encoded bitstream.

[0155] Decoder 800 can decode the coded bitstream EB output from selection circuit 701 or the coded bitstream output from encoder 700 and generate reconstructed image data or reconstructed video data RV. The reconstructed image data or reconstructed video data RV can be displayed on display 40 through display interface 260.

[0156] In some exemplary embodiments, the encoder 700 or the decoder 800 may store the reconstructed image data or the reconstructed video data RV in the memory MEM1. The encoder 700 or the decoder 800 may perform in-loop filtering of the stored reconstructed image data or the stored reconstructed video data in the memory MEM1.

[0157] In some example embodiments, each of the encoder 700 and the decoder 800 may include in-loop filtering defined according to an H.264 video coding standard or HEVC video coding standard or any other standard that supports an in-loop deblocking filter.

[0158] Fig. 16 is a block diagram of an encoder used in Fig. 15 is shown.

[0159] With reference to the Fig. 15 and Fig. 16, the encoder 700 may include a predictor 710, a subtractor 720, a converter 730, a quantizer 740, an entropy encoder 750, an inverse quantizer 760, an inverse converter 770, an adder 780, and an in-loop filter 790.

[0160] Predictor 710 may include a motion estimator 712 and a motion compensator 714. Predictor 710 may predict a current block, generate a predicted block, and output the predicted block. Specifically, predictor 710 may predict a pixel value of each of the pixels in the current block and generate a predicted block including the predicted pixel value of each of the pixels. When performing inter-prediction, predictor 710 may include motion estimator 712 and motion compensator 714. Motion estimator 712 may be called a motion vector estimator.

[0161] The motion estimator 712 or the motion vector estimator 712 may estimate a motion vector of a current block based on at least one reference image through a block corresponding to a block mode or a predefined block mode of the current block, and determine the motion vector of the current block. A block size may be 16×16, 16×8, 8×16, 8×8, 8×4, 4×8, or 4×4.

[0162] The motion compensator 714 may generate a predicted block based on the motion estimation vector of the current block output from the motion estimator 712 and output the predicted block. That is, the motion compensator 714 may output a block in the reference image addressed by the current block as a predicted block.

[0163] Subtractor 720 may subtract the predicted block from the current block having video data INV and generate a remaining block. Subtractor 720 may calculate a difference between each pixel of the current block and each pixel of the predicted block and generate the remaining block having remaining signals.

[0164] Converter 730 may convert the remaining signals from a time domain to a frequency domain, and quantizer 740 may quantize an output of converter 730 and output the quantized remaining block. Converter 730 may use a discrete cosine transform (DCT) to transform the remaining signal into the frequency domain. The remaining signals converted into the frequency domain may be a transform coefficient.

[0165] The entropy encoder 750 may encode the quantized remaining block output from the quantizer 740 and output an encoded bitstream.

[0166] The inverse converter 770 may perform an inverse transformation of an inverse quantized remaining block and generate a reconstructed remaining block.

[0167] The adder 780 may add the predicted block from the predictor 710 and the reconstructed remaining block output from the inverse converter 770, and reconstruct the current block.

[0168] The in-loop filter 790 may perform in-loop filtering of the reconstructed current block and output the filtered current block to the predictor 710.

[0169] Fig. 17 is a block diagram of a decoder used in Fig. 15 is shown.

[0170] With reference to the Fig. 15 to 17, the decoder 800A may include an entropy decoder 810A, an inverse quantizer 820, an inverse converter 830, an adder 850, and an in-loop filter 860A.

[0171] The entropy decoder 810A may receive a received coded bit stream EBS or a coded bit stream output from the encoder 700 and generate a quantized remaining block.

[0172] The inverse quantizer 820 may perform inverse quantization of the quantized remaining block output from the entropy decoder 810A and generate an inverse quantized remaining block.

[0173] The inverse converter 830 may perform an inverse conversion of the inverse quantized remaining block and generate a reconstructed remaining block.

[0174] The motion compensator 840 may predict a current block based on a motion vector included in the reconstructed block output from the in-loop filter 860A and output the predicted block.

[0175] The adder 850 may reconstruct the current block by adding the reconstructed remaining block output from the in-loop filter 860 and the predicted block output from the motion compensator 840A, and output the reconstructed current block.

[0176] The in-loop filter 860A can perform in-loop filtering of the reconstructed current block and output the reconstructed current block RV. That is, the in-loop filter 860A can output the reconstructed video data RV by one block.

[0177] In some example embodiments, the in-loop filters 790 and 860A may each include a deblocking filter and a Sample Adaptive Offset (SAO) filter.

[0178] The Fig. 18 and Fig. 22 are conceptual diagrams illustrating a method of processing video data by modifying motion compensation information according to example embodiments.

[0179] With reference to the Fig. 3, 4, 7, 8 and 18 to 22, each image 610, 612, 614, 630, 632 or 634 may include a divided region which may be independently encoded or decoded. The divided region may include at least one slice. It is noted that in the following description of the Fig. 18 to 22 not all images and discs will be described for the sake of brevity. For example, images 616 ( Fig. 3), 636 and 638 ( Fig. 7) and the slice SLICE1-3 ( Fig. 7) is not explicitly described here. In addition, the method for processing video data, which is described in the Fig. 18 to 22, is described with respect to two panes. However, the method is applicable to exemplary embodiments relating to tiles.

[0180] A first image 610 or 630 may include a first slice SLICE1-1 and a second slice SLICE1-2, a second image 612 and 632 may include a third slice SLICE2-1 and a fourth slice SLICE2-2, and a third image 614 and 634 may include a fifth slice SLICE3-1 and a sixth slice SLICE3-2.

[0181] As in Fig. 18, each of the fifth slice SLICE3-1, the third slice SLICE2-1, and the fourth slice SLICE2-2 may each include a fifth macroblock M5, a third macroblock M3, and a fourth macroblock M4.

[0182] If the third image 614 or 634 is generated by the first core COREO (in Fig. 4) or the third core CORE2 (in Fig. 8), the decoder 800 or 800A may extract first motion compensation information, or first motion compensation information, from an encoded bitstream corresponding to the fifth macroblock M5. The first motion compensation may include a reference image index and / or a reference coordinate. The first motion compensation information may include reference information, such as a reference index or a motion vector, that satisfies a motion processing standard specification, such as H.264 or HEVC.

[0183] Referring to Fig. 18, if the first motion compensation information of the fifth macroblock M5 in the fifth slice SLICE3-1 refers to a coordinate in the fourth slice SLICE2-2 of the second image 612, which is decoded in parallel with the fifth slice SLICE3-1, a decoding operation of the fourth slice SLICE2-2 may not be completed. Therefore, the data processing system 100A may modify the first compensation information, for example, a reference coordinate, to avoid a decoding dependency between the fourth macroblock M4 in the fourth slice SLICE2-2 and the fifth macroblock M5 in the fifth slice SLICE3-1.

[0184] For example, data processing system 100A may modify an X-axis value while maintaining a Y-axis value in the reference coordinate of the first motion compensation information. That is, data processing system 100A may modify the reference coordinate to refer to the third macroblock M3 in the bottom of the third slice SLICE2-1 for decoding the fifth macroblock M5. Thus, data processing system 100A may decode the fifth slice SLICE3-1 by referring to the third macroblock M3 in the third slice SLICE2-1 instead of the fourth macroblock M4 in the fourth slice SLICE2-2.

[0185] By modifying the motion compensation information for decoding a macroblock, there may be an error in the decoding result of the macroblock. A method for reducing the decoding error caused by modifying the motion compensation information will be described later with reference to Fig. 23A and Fig. 23B.

[0186] Referring to Fig. 19 can, as in Fig. 18, if a first piece of motion compensation information of the fifth macroblock M5 in the fifth slice SLICE3-1 refers to a coordinate in the fourth slice SLICE2-2 of the second image 612 decoded in parallel with the fifth slice SLICE3-1, a decoding operation of the fourth slice SLICE2-2 may not be completed. Accordingly, the data processing system 100A may modify the first piece of compensation information, for example, a reference image index, to avoid a decoding dependency between the fourth macroblock M4 in the fourth slice SLICE2-2 and the fifth macroblock M5 in the fifth slice SLICE3-1.

[0187] For example, data processing system 100A may modify the reference image index of the first motion compensation information while maintaining a reference coordinate of the first motion compensation information. That is, data processing system 100A may modify the reference image index to refer to the second macroblock M2 in the second slice SLICE1-2 for decoding the fifth macroblock M5, as shown in Fig. 19. Thus, the data processing system 100A may decode the fifth slice SLICE3-1 by referring to the second macroblock M2 in the second slice SLICE1-2 instead of the fourth macroblock M4 in the fourth slice SLICE2-2.

[0188] Referring to Fig. 20, the third image 614 may be an image displayed before the second image 612 and decoded after the second image 612. In such a case, first motion compensation information of the fifth macroblock M5 in the fifth slice SLICE3-1 may refer to the first image 610 and the second image 612 in a bidirectional manner as reference images. If the fifth macroblock M5 in the fifth slice SLICE3-1 refers to the first macroblock M1 in the first slice SLICE1-1 and the fourth macroblock M4 in the fourth macroblock in the fourth slice SLICE2-2, a decoding operation of the fourth slice SLICE2-2 may not be completed. Accordingly, the data processing system 100A may modify the first compensation information, for example, reference image indices, to avoid a decoding dependency between the fourth macroblock M4 in the fourth slice SLICE2-2 and the fifth macroblock M5 in the fifth slice SLICE3-1.

[0189] For example, data processing system 100A may modify the reference image indices of the first motion compensation information. This means that data processing system 100A may modify the first compensation information from a bidirectional reference to a unidirectional reference. Accordingly, data processing system 100A may decode the fifth macroblock M5 in the fifth slice SLICE3-1 by referring only to the first macroblock M1 in the first slice SLICE1-1, without referring to the fourth macroblock M4 in the fourth slice SLICE2-2.

[0190] Referring to Fig. 21, the third picture 614 may be a picture displayed before the second picture 612 and decoded after the second picture 612. The first motion compensation information of the fifth macroblock M5 in the fifth slice SLICE3-1 may refer to the first picture 610 and the second picture 612 in a bidirectional manner as reference pictures. When the fifth macroblock M5 in the fifth slice SLICE3-1 refers to the first macroblock M1 in the first slice SLICE1-1 and the fourth macroblock M4 in the fourth macroblock in the fourth slice SLICE2-2, a decoding operation of the fourth slice SLICE2-2 may not be completed. Accordingly, the data processing system 100A may modify the first compensation information, for example, reference image indices and a reference coordinate, to avoid a decoding dependency between the fourth macroblock M4 in the fourth slice SLICE2-2 and the fifth macroblock M5 in the fifth slice SLICE3-1.

[0191] For example, data processing system 100A may modify the reference frame indices and the reference coordinate of the first motion compensation information to refer to the third macroblock M3 in the third slice SLICE2-1 (1) or the second macroblock M2 in the second slice SLICE1-2 (2). That is, data processing system 100A may modify the first compensation information by maintaining the bidirectional reference. Accordingly, data processing system 100A may decode the fifth macroblock M5 in the fifth slice SLICE3-1 by referring to the first macroblock M1 in the first slice SLICE1-1 and one of the third macroblock M3 in the third slice SLICE2-1 or the second macroblock M2 in the second slice SLICE1-2.

[0192] Referring to Fig. 22, if the first motion compensation information of the fifth macroblock M5 in the fifth slice SLICE3-1 relates to the fourth block M4 in the fourth macroblock in the fourth slice SLICE2-2, a decoding operation of the fourth slice SLICE2-2 may not be completed. Therefore, the data processing system 100A may modify the first compensation information, for example, a reference image index and a reference coordinate, to avoid decoding dependency between the fourth macroblock M4 in the fourth slice SLICE2-2 and the fifth macroblock M5 in the fifth slice SLICE3-1.

[0193] For example, data processing system 100A may modify the reference frame index of the first motion compensation information with the same reference coordinate to refer to the second macroblock M2 in the second slice SLICE1-2 and the third macroblock M3 in the third slice SLICE2-1. This means that data processing system 100A may modify the first compensation information from a unidirectional reference to a bidirectional reference. Accordingly, data processing system 100A may decode the fifth macroblock M5 in the fifth slice SLICE3-1 by referring to the second macroblock M2 in the second slice SLICE1-2 and the third macroblock M3 in the third slice SLICE2-1.

[0194] For the purpose of easy understanding, the Fig. 18 to 22, the fifth macroblock M5 refers to other macroblocks, for example, M1, M2 or M3, etc. However, a reference block (for example, M1, M2 or M3, etc.) referred to by the fifth macroblock M5 may be any block other than a macroblock according to a video coding standard, and a block size of the reference block (for example, M1, M2 or M3, etc.) referred to by the fifth block M5 may be the same as or different from the block size of the fifth block M5.

[0195] For example, if the block size of the fifth macroblock M5 is 16 by 16, the block size of a reference block (e.g., M1, M2, or M3, etc.) referred to by the fifth macroblock M5 may be 4 by 4. Therefore, the reference block referred to by the fifth macroblock M5 may not always be a macroblock having the same block size, and may be a subblock smaller than the macroblock or any combination of subblocks. For example, assuming in Fig. 18, that the block size of the fifth macroblock M5 is 16×16, the motion compensation information of the fifth macroblock M5 may be modified to refer to the third macroblock M3 whose size is 4×4.

[0196] Additionally, the motion compensation information of a macroblock may correspond to a unit of a macroblock or a unit of a sub-block smaller than the unit of the macroblock. Thus, a piece of motion compensation information corresponding to a macroblock may include a plurality of motion compensation information corresponding to a plurality of sub-blocks included in the macroblock.

[0197] As in the method for processing video data by modifying motion compensation information in the Fig. 18 to 22, the divided region in each image 610, 612, 614, 630, 632, or 634 may include at least one slice. However, as discussed above, the method may also be applied to tiles. In the case where the divided region includes a plurality of tiles, the data processing system 100A may modify motion compensation information by changing at least one of an X-axis value and a Y-axis value of a reference coordinate when the reference coordinate is changed in a reference image according to a divided format of the plurality of tiles.

[0198] With reference to the Fig. 18 to 22, the method for processing video data by data processing system 100A may modify the motion compensation information according to a number of reference frames, a value of a motion vector, and / or a boundary matching error estimate. The boundary matching error estimate refers to a difference in pixel values ​​at the boundary between adjacent decoded blocks prior to a deblocking operation.

[0199] For example, data processing system 100A may modify the motion compensation information of a current decoding block selected for decoding to refer to the most recent previous frame for a reference frame when there are a plurality of reference frame candidates. On the other hand, data processing system 100A may modify the motion compensation information of the current decoding block to refer to a reference frame or reference coordinate (or motion vector) that can make the motion vector smallest.Alternatively, when a plurality of macroblock or subblock candidates can be referenced for decoding the current decoding block, the data processing system 100A may modify the motion compensation information of the current decoding block to refer to one of the macroblock or subblock candidates that has a minimum difference in pixel values ​​at the boundary between adjacent decoded macroblocks or adjacent decoded subblocks before a deblocking operation. Here, the pixel value may indicate pixel data and may have a format according to one of data formats such as RGB, YUV, or YcbCr, etc.

[0200] The method for processing video data, which is used in the Fig. 18 to 22 may cause errors in pixel values ​​of a constructed macroblock because the modification of a motion compensation vector of a current decoding block may cause a reference image or reference coordinate (or motion vector) defined by encoding the video data to be altered, thus causing the current decoding block to refer to a different macroblock than an original macroblock defined by original motion compensation information before generating modified motion compensation information. Thus, the pixels in the constructed macroblock may have pixel errors even if the data processing system assigns a macroblock having similar pixel values ​​to an original reference macroblock.

[0201] Errors in pixel values ​​can cause unnatural boundary displays at the boundaries between reconstructed blocks. Here, errors in pixel values ​​in the reconstructed block can be spread across other blocks adjacent to the reconstructed block by performing in-loop filtering at the block boundaries.

[0202] According to an exemplary embodiment, the errors in pixel values ​​of the reconstructed blocks, which are generated by modifying the motion compensation information of the reconstructed block, may be prohibited from spreading across other neighboring blocks at the boundaries of the reconstructed blocks. Partial in-loop filtering, which addresses the unfavorable distribution of errors at boundaries, will be described in detail later with reference to Fig. 23A and Fig. 23B.

[0203] Although the method for processing video data by modifying motion compensation information described in the Fig. 18 to 22 illustrates that each image 610, 612, and 614 comprises a plurality of slices, each of which is a processing unit. The method for processing video data can be applied to an image 610, 612, and 614 having only one divided region, and in particular, only one slice. Additionally, the method for processing video data can be used by modifying motion compensation information when a plurality of divided regions in an image, e.g., slices or tiles, are decoded by a plurality of cores.

[0204] For example, the data processing system 100A may modify first motion compensation information of current decoding blocks in a picture having only one slice to cause the current decoding block to refer to an area in a reference picture in which a decoding operation is finished, even in the case where the entire area of ​​the reference picture is not yet decoded, or to cause the current decoding blocks to refer to a picture whose decoding is finished.

[0205] The Fig. 23A and Fig. 23B are conceptual diagrams illustrating partial in-loop filtering according to an example embodiment.

[0206] Fig. Figure 23A is a conceptual diagram illustrating in-loop filtering performed between adjacent blocks. A macroblock M1 can perform in-loop filtering by referring to pixel values ​​of the adjacent blocks at the block boundaries, as indicated by the double-headed arrows in Fig. 23A is illustrated.

[0207] Fig. 23B is a conceptual diagram illustrating partial in-loop filtering performed for a reconstructed block by modifying first motion compensation information of the reconstructed block. Referring to Fig. 23B, a method for error compensation according to an exemplary embodiment may perform in-loop filtering for the current macroblock M1, but omit in-loop filtering for neighboring macroblocks of the current macroblock M1. The in-loop filtering for the current macroblock M1 may be performed only for pixels at the boundaries of the current macroblock M1. The restricted in-loop filtering described above may be called "partial in-loop filtering." The partial in-loop filtering may perform in-loop filtering for only a portion of the boundaries of the current macroblock M1.

[0208] For example, data processing system 100A may prevent the errors of pixel values ​​of a current decoding block M1 from being propagated across neighboring blocks at the boundaries of the current decoding block M1 by performing in-loop filtering for only internal pixels at the boundary of the current decoding block M1. Data processing system 100A may perform a deblocking effect at the boundaries of the current decoding block M1 by performing partial in-loop filtering according to the present inventive concept.

[0209] An in-loop filter that can perform partial in-loop filtering can perform partial in-loop filtering that can be Fig. 23B when a received first control signal is activated, and may perform normal in-loop filtering, which is described in Fig. 23A when the received first control signal is deactivated.

[0210] Fig. 24 is a block diagram illustrating a decoder according to an exemplary embodiment. Referring to the Fig. 24, a decoder 800B may include an entropy decoder 810B, an inverse quantizer 820, an inverse converter 830, a motion compensator 840B, an adder 850, and an in-loop filter 860B that performs partial in-loop filtering. The decoder 800B may be incorporated into at least one of the hardware codecs 230-1 to 230-N shown in Fig. 1 are illustrated.

[0211] As in the Fig. 18 to 23B, decoder 800B may provide a first control signal IIL to in-loop filter 860B when decoder 800B modifies the motion compensation information of a current decoding block. As discussed above, a state of first control signal IIL may determine partial in-loop filtering or normal in-loop filtering.

[0212] The first control signal can be obtained from the entropy decoder 810B as shown in Fig. 24 or provided by motion compensator 840B. In such a case, entropy decoder 810B or motion compensator 840B may include a motion compensation information modifier 870. For example, when decoder 800B receives an encoded bitstream, entropy decoder 810B may extract first motion compensation information for decoding a macroblock from the received encoded bitstream. The first motion compensation information may include a reference image and a reference coordinate of the decoding macroblock or subblocks of the decoding macroblock.

[0213] Entropy decoder 810B may generate decoding history information indicating whether a decoding operation of a picture, slice, or tile included in the encoded bitstream has been completed. The decoding history information may be stored in a buffer or cache memory in entropy decoder 810B or decoder 800B. The decoding history information may include digital bits, each of which corresponds to a decoding result of the picture, slice, or tile by entropy decoder 810B or decoder 800B.

[0214] The entropy decoder 810B may determine whether the first motion compensation information should be modified to increase parallel processing performance based on the first motion compensation information and the decoding history information, and may provide a first control signal IIL to the in-loop filter 860B when the first motion compensation information is modified according to the determination result.

[0215] The modified motion compensation information and the first control signal IIL may be provided by the motion compensator 840B instead of the entropy decoder 810B. Hereinafter, for convenience in explanation, duplicate explanations of the same structure and operations as those according to the above-described exemplary embodiments will be omitted, and explanations will be made about different points between the exemplary embodiments.

[0216] The decoding history information can be stored in the memories MEM1 to MEMn in Fig. 1 and shared by the hardware codecs 230-1 through 230-N or may be stored in a memory of the data processing system 100A.

[0217] Fig. 25 is a block diagram illustrating a data processing system according to an exemplary embodiment. Referring to Fig. 25, a data processing system 900 may include a first core 230-1, a second core 230-2, a decoding history storage device 910, and a motion compensation information modifier 920.

[0218] The decoding history storage device 910 and the motion compensation information modifier 920 may be shared by each of the first and second cores 230-1 and 230-2, while the decoder 800B in Fig. 24 has the decoding history memory and the motion compensation information modifier 870 which are provided independently.

[0219] The decoding history storage device 910 may store decoding history information indicating whether or not the decoding of an image, a slice, and / or a tile decoded by each of the first and second cores 230-1 and 230-2 is completed.

[0220] The motion compensation information modifier 920 may receive motion compensation information of a current decoding macroblock from each of the first and second cores 230-1 and 230-2, and may provide a first control signal IIL for partial in-loop filtering and modified motion compensation information for each of the first and second cores 230-1 and 230-2 based on the received motion compensation information and the decoding history information stored in the decoding history storage device 910.

[0221] Although Fig. 25 illustrates two cores 230-1 and 230-2, the decoding history storage device 910 and the motion compensation information modifier 920 may be shared by more than two cores.

[0222] In some example embodiments, the data processing system 900 may further include an in-loop filter 930 shared by the first and second cores 230-1 and 230-2. When the first and second cores 230-1 and 230-2 transmit data of a decoded macroblock to the in-loop filter 930 for in-loop filtering, the in-loop filter 930 may transmit data of the macroblock filtered by the in-loop filter 930 to the first and second cores 230-1 and 230-2.

[0223] Here, the in-loop filter 930 may perform partial in-loop filtering of a current decoding block in response to the first control signal IIL received from the motion compensation information modifier 920. When the first control signal IIL is activated, the in-loop filter 930 may perform partial in-loop filtering of the current decoding block and transmit data corresponding to the macroblock filtered by the partial in-loop filtering to the first and second cores 230-1 and 230-2. When the first control signal IIL is deactivated, the in-loop filter 930 may perform normal in-loop filtering.

[0224] Fig. 26 is a data format illustrating an encoded bitstream including compensation information according to exemplary embodiments. Referring to Fig. 26, a portion of an encoded bitstream 1000 may be associated with motion compensation information 1010, which includes a block type 1010-1, a reference frame index 1010-2, and a reference X-coordinate 1010-3 and a reference Y-coordinate 1010-4. In some exemplary embodiments, block type 1010-1 may be omitted.

[0225] As in the Fig. 20 to 22, when a decoding block bidirectionally relates to more than one picture, the motion compensation information 1010 may include the block type 1010-1 indicating a macroblock type, a reference picture index 1010-2 corresponding to the block type 1010-1, and a reference coordinate including an X coordinate 1010-3 and a Y coordinate 1010-4, but the motion compensation information 1010 is not limited to this. Alternatively, when the decoding block does not bidirectionally relate to more than one picture, the motion compensation information 1010 may omit the block type 1010-1.

[0226] A syntax related to the reference picture and the reference coordinate may vary according to a related video standard specification. For example, the syntax related to the motion compensation information according to the H.264 standard may include a reference picture ID ref_idx_lx indicating a reference picture of the current decoding block and a differential motion vector mvd_lx indicating a coordinate difference between the current decoding block and a reference block of the current decoding block.

[0227] Additionally, the syntax may include a macroblock type mb_type indicating a unidirectional reference or a bidirectional reference. The reference image ID ref_idx_lx or the differential motion vector mvd_lx with respect to the Fig. 18 to 22 can be modified, and the macroblock type mb_type can be modified with reference to the Fig. 20 and Fig. 22. The macroblock type mb_type can be assigned to the block type 1010-1 in Fig. 26 and the reference image ID ref_idx_lx can be assigned to the reference image index 1010-2 in Fig. 26. The differential motion vector mvd_lx can be assigned to the X coordinate 1010-3 and the Y coordinate 1010-4 in Fig. 26 correspond.

[0228] Fig. 27 is a block diagram illustrating a data processing system according to an exemplary embodiment.

[0229] Referring to Fig. 27, a data processing system 100B may include a controller 200B, a plurality of memories 301, 303, 305, and a display 400. In some example embodiments, the data processing system 100B may further include a camera 500.

[0230] The controller 200B may be implemented in an integrated circuit (IC), a system-on-chip (SoC), an application processor (AP), and / or a mobile AP, etc.

[0231] The controller 200B may include a CPU 210A, a GPU 220, a plurality of hardware codecs 230, a storage interface (I / F) 240, a display interface (I / F) 260, and a receiver interface (I / F) 270. If the data processing system 100A further includes a camera 500, the controller 200A may further include a camera interface (I / F) 250.

[0232] The controller 200B, which is Fig. 27 is equal to or similar to the controller 200A shown in Fig. 1 in terms of structure and operation except for the plurality of codecs 231-1 to 231-N which are included in a hardware codec.

[0233] The operation of each of the plurality of codecs 231-1 to 231-N may be the same as or similar to the operation of each of the plurality of hardware codecs 230-1 to 230-N in Fig. 1. In addition, each of the plurality of codecs 231-1 to 231-N may have a memory corresponding to each of the memories MEM1 to MEMN.

[0234] Each of the plurality of codecs 231-1 to 231-N may include components 701, 700 and 800, which may be described with reference to the Fig. 15 to 17. The first core CORE0, the second core CORE1 and the third core CORE2, which are described in the Fig. 4 to Fig. 14 may each correspond to a first core 231-1, a second core 231-2, and a third core 231-3.

[0235] Fig. 28 is a block diagram illustrating a data processing system according to an exemplary embodiment.

[0236] Referring to Fig. 28, a data processing system 100C may include a controller 200C, a plurality of memories 301, 303, 305, and a display 400. In some example embodiments, the data processing system 100C may further include a camera 500.

[0237] The controller 200C may be implemented in an integrated circuit (IC), a system on chip (SoC), an application processor (AP), and / or a mobile AP, etc.

[0238] The controller 200C may include a CPU 210B, a GPU 220, a plurality of hardware codecs 230, a memory interface (I / F) 240, a display interface (I / F) 260, and a receiver interface (I / F) 270. If the data processing system 100C further includes a camera 500, the controller 200C may further include a camera interface (I / F) 250.

[0239] The 200C controller, which is Fig. 27 is equal to or similar to the controller 200A shown in Fig. 1, in terms of structure and operation, except that the CPU 210B has a plurality of cores CORE1 to COREN.

[0240] The operation of each of the plurality of cores CORE1 to COREN may be the same as or similar to the operation of each of the plurality of hardware codecs 230-1 to 230-N in Fig. 1. In addition, each of the plurality of codecs 231-1 to 231-N may have a memory (e.g., a cache) corresponding to each of the memories MEM1 to MEMN.

[0241] Each of the plurality of cores CORE1 to COREN can execute a software codec. The first core CORE0, the second core CORE1 and the third core CORE2, which are located in the Fig. 4 to Fig. 14, a first core CORE1, a second core CORE2 and a third core CORE2 in Fig. 28 correspond each time N is three.

[0242] The software codec executed by each of the cores CORE1 to COREN may perform encoding, decoding, encoding including in-loop filtering, or decoding including in-loop filtering with reference to the Fig. Perform steps 4 to 14.

[0243] Fig. 29 is a flowchart illustrating a method for processing video data by the data processing system incorporated in the Fig. 1, Fig. 18 and Fig. 19 according to an exemplary embodiment. The method of processing may be, for example, encoding or decoding.

[0244] With reference to the Fig. 1 to 20, in operation S100, each image is assigned to a core. For example, each of a first image, a second image, and a third image in video data may be assigned to one of a first core and a second core for processing the video data. Each of the first image, the second image, and the third image may each include a first slice (or a first tile) and a second slice (or a second tile).

[0245] In operation S110, a first slice (or a first tile) in the first image may be processed by the first core. In operation S120, a first slice (or a first tile) in the second image may be processed in parallel by the second core while a second slice (or a second tile) of the first image is processed by the first core. In operation S120, the first core may further perform in-loop filtering of the first slice and the second slice of the first image based on the processing result of the first slice of the first image. In operation S130, a first slice (or a first tile) in the third image may be processed in parallel by the third core while a second slice (or a second tile) of the second image is processed by the second core.In operation S130, the second core may further perform in-loop filtering of the first slice and the second slice of the second image based on the processing result of the first slice of the second image.

[0246] As described above, the method for decoding video data described in the Fig. 1 to 14 and 18 to 20 may similarly be used for a method of encoding the video data by allocating each of a plurality of pictures to each of a plurality of cores and processing the plurality of pictures in parallel by the plurality of cores.

[0247] Fig.30 is a flowchart illustrating a method for processing video data according to another exemplary embodiment. In operation S210, decoding history information of each core may be stored. In operation S220, motion compensation information of a current decoding block may be extracted. In operation S230, the motion compensation information may be modified. In operation S240, the current block may be decoded based on the modified motion compensation information. In operation S250, internal in-loop filtering may be performed.

Claims

[1] Data processing system (100A, 100B, 100C) comprising: a first core (CORE0) configured to decode a first image (10, 610, 620, 630, 640, 650, 660) and to generate decoding history information of the decoded first image (10, 610, 620, 630, 640, 650, 660); and a second core (CORE1) configured to modify first motion compensation information (1010) contained in a second image (612, 622, 632, 642, 652, 662) based on the decoding history information of the decoded first image (10, 610, 620, 630, 640, 650, 660), and to decode the second image (612, 622, 632, 642, 652, 662) based on the decoded first image (10, 610, 620, 630, 640, 650, 660) and the modified first motion compensation information (1010), wherein the second core (CORE1) is configured to generate the decoding history information of the decoded second image (612, 622, 632, 642, 652, 662), and the first core (COREO) is configured to modify second motion compensation information (1010) contained in a third image (614, 624, 634, 644, 654, 664) based on the decoding history information of the decoded second image (612, 622, 632, 642, 652, 662), and to modify the third image (614, 624, 634, 644, 654, 664) based on the modified second motion compensation information (1010) and one of the decoded first image (610, 620, 630, 640, 650, 660) and the decoded second image (612, 622, 632, 642, 652, 662) to decode, wherein the first core (CORE0) comprises a first motion compensation information modifier (870, 920) configured to modify the second motion compensation information (1010), and the second core (CORE1) comprises a second motion compensation information modifier (870, 920) configured to modify the first motion compensation information (1010), wherein the first core (CORE0) has a first in-loop filter (790, 860A, 860B, 930) connected to the first motion compensation information modifier (870, 920), and the second core (CORE1) has a second in-loop filter (790, 860A, 860B, 930) connected to the second motion compensation information modifier (870, 920), and wherein the first in-loop filter (790, 860A, 860B, 930) is configured to perform partial in-loop filtering, which is a deblocking operation only in a block related to the modified first motion compensation information (1010), based on a first control signal received from the first motion compensation information modifier (870, 920), and the second in-loop filter (790, 860A, 860B, 930) is configured to perform partial in-loop filtering, which is a deblocking operation only in a block related to the modified second motion compensation information (1010), based on a second control signal received from the second motion compensation information modifier (870, 920). [2] The data processing system (100A, 100B, 100C) of claim 1, wherein each of the first image (10, 610, 620, 630, 640, 650, 660), the second image (612, 622, 632, 642, 652, 662) and the third image (614, 624, 634, 644, 654, 664) includes at least one divided region. [3] The data processing system (100A, 100B, 100C) according to claim 1, wherein the first image (10, 610, 620, 630, 640, 650, 660) has a first divided region, the decoding history information of the decoded first image (10, 610, 620, 630, 640, 650, 660) has decoding history information of the first divided region, and the second core (CORE1) modifies the first motion compensation information (1010) based on the decoding history information of the first divided region. [4] The data processing system (100A, 100B, 100C) according to claim 3, wherein the modified first motion compensation information (1010) comprises a modified reference coordinate corresponding to a reference coordinate included in the first motion compensation information (1010). [5] The data processing system (100A, 100B, 100C) of claim 3, wherein the second image (612, 622, 632, 642, 652, 662) includes a second divided region, the decoding history information of the decoded second image (612, 622, 632, 642, 652, 662) includes decoding history information of the second divided region, and the first core (COREO) modifies the second motion compensation information (1010) based on the decoding history information of the second divided region. [6] The data processing system (100A, 100B, 100C) according to claim 5, wherein the modified second motion compensation information (1010) comprises a modified reference image index (1010-2) modified with respect to a reference image index (1010-2) included in the second motion compensation information (1010). [7] A method for decoding a plurality of images in video data, the method comprising: receiving a first image (10, 610, 620, 630, 640, 650, 660) having a first divided region; receiving a second image (612, 622, 632, 642, 652, 662) having a second divided region; extracting first motion compensation information (1010) contained in the second image (612, 622, 632, 642, 652, 662); modifying the extracted first motion compensation information (1010); and decoding the second image (612, 622, 632, 642, 652, 662) based on the first divided area and the modified first motion compensation information (1010), wherein modifying the extracted first motion compensation information (1010) comprises modifying a reference coordinate of a reference image included in the first motion compensation information (1010) based on decoding history information of the reference image related to the extracted first motion compensation information (1010), wherein the first image (10, 610, 620, 630, 640, 650, 660) has a third divided region different from the first divided region, and modifying the reference coordinate of the reference image comprises changing the reference coordinate from a first coordinate relating to the first divided area to a second coordinate relating to the third divided area, and the method further comprising: decoding a block contained in the second divided area based on the first image (10, 610, 620, 630, 640, 650, 660) and the modified first motion compensation information (1010); and performing partial in-loop filtering of the decoded block. [8] A method for decoding a plurality of images in video data, the method comprising: receiving a first image (10, 610, 620, 630, 640, 650, 660) having a first divided region; receiving a second image (612, 622, 632, 642, 652, 662) having a second divided region; extracting first motion compensation information (1010) contained in the second image (612, 622, 632, 642, 652, 662); modifying the extracted first motion compensation information (1010); decoding the second image (612, 622, 632, 642, 652, 662) based on the first divided area and the modified first motion compensation information (1010); receiving a third image (614, 624, 634, 644, 654, 664) having a third divided region; extracting second motion compensation information (1010) contained in the third image (614, 624, 634, 644, 654, 664); modifying the extracted second motion compensation information (1010); and decoding the third image (614, 624, 634, 644, 654, 664) based on the modified second motion compensation information (1010), wherein modifying the extracted second motion compensation information (1010) comprises modifying a reference picture index (1010-2) included in the second motion compensation information (1010) based on decoding history information of a reference picture referred to by the reference picture index (1010-2), and wherein modifying the reference image index (1010-2) comprises changing the reference image index (1010-2) from a second index relating to the second image (612, 622, 632, 642, 652, 662) to a first index relating to the first image (10, 610, 620, 630, 640, 650, 660), and the method further comprising: decoding a block in the third divided area based on the second image (612, 622, 632, 642, 652, 662) and the modified second motion compensation information (1010); and performing partial in-loop filtering of the decoded block. [9] The method according to claim 8, wherein the extracted second motion compensation information (1010) includes a reference picture index (1010-2); and modifying the extracted second motion compensation information (1010) comprises modifying the reference picture index (1010-2) and a reference coordinate of a reference picture referred to by the reference picture index (1010-2) included in the extracted second motion compensation information (1010) based on decoding history information of the first divided area or the second divided area related to the reference picture index (1010-2) and the reference coordinate of the reference picture. [10] The method of claim 9, wherein the first image (10, 610, 620, 630, 640, 650, 660) has a fourth divided region different from the first divided region, and modifying the extracted first motion compensation information (1010) comprises: modifying the reference image index (1010-2) such that the reference image index (1010-2) is changed from a second index relating to the second image (612, 622, 632, 642, 652, 662) to a first index relating to the first image (10, 610, 620, 630, 640, 650, 660); and modifying the reference coordinate of the reference image such that the reference coordinate is changed from a first coordinate relating to the fourth divided region to a second coordinate relating to the first divided region.

Images