Encoding apparatus for parallel encoding and method for encoding image by using same, and decoding apparatus and method for decoding image by using same

Abstract

A method by which an encoding apparatus encodes an image may comprise the steps of: identifying a plurality of segments from an image; encoding the plurality of segments in parallel; storing substreams generated through parallel encoding in a plurality of buffers; while the substreams are stored in the plurality of buffers, if the current state corresponds to a preset output condition, acquiring substream packets having the predetermined size from the plurality of buffers; and arranging the substream packets in a predetermined order and outputting a bitstream in which the substream packets are arranged.

Description

Encoding device for parallel encoding and method for encoding an image using the same, and decoding device and method for decoding an image using the same

[0001] The present disclosure relates to the field of image encoding and decoding, and more particularly, the present disclosure relates to a device and method for encoding and decoding an image using parallel processing technology.

[0002] In image encoding and decoding, the image is divided into blocks, and each block can be predicted and decoded through inter prediction or intra prediction.

[0003] Inter prediction is a technique for compressing images by removing temporal redundancy between images. Inter prediction allows blocks in the current image to be predicted using a reference image. The reference block most similar to the current block can be found within the reference image. The current block is predicted based on the reference block, and the resulting predicted block is subtracted from the current block to generate a residual block.

[0004] Intra prediction is a technique for compressing images by removing spatial redundancy within the image. Intra prediction generates a predicted block based on the surrounding pixels of the current block, depending on the intra prediction mode. Then, the predicted block is subtracted from the current block to generate a residual block.

[0005] Information about residual blocks can be included in the bitstream. The decoder can obtain the residual block from the bitstream and reconstruct the current block using the residual block obtained from the bitstream and the predicted block generated through inter-prediction or intra-prediction.

[0006] As multi-core CPUs become more common in mobile devices, standards like HEVC have adopted parallelization techniques. Parallelization involves dividing data to be processed by a program into multiple units and then assigning these units to different cores or threads to perform the same task in parallel. Parallel encoding and decoding of video can reduce the time required for encoding and decoding.

[0007] A method of encoding an image according to one embodiment may include a step of identifying a plurality of segments from the image.

[0008] A method of encoding an image according to one embodiment may include a step of encoding a plurality of segments in parallel.

[0009] A method for encoding an image according to one embodiment may include a step of storing substreams generated through parallel encoding in a plurality of buffers.

[0010] A method of encoding a video according to one embodiment may include a step of obtaining substream packets having a predetermined size from a plurality of buffers when a current state corresponds to a preset output condition while the substreams are stored in a plurality of buffers.

[0011] A method for encoding an image according to one embodiment may include the steps of arranging substream packets in a predetermined order and outputting a bitstream in which the substream packets are arranged.

[0012] A method of decoding an image according to one embodiment may include a step of identifying a plurality of segments within the image.

[0013] A method for decoding an image according to one embodiment may include obtaining substream packets of a predetermined size arranged in a predetermined order from a bitstream. In one embodiment, the substream packets may be generated through parallel encoding of a plurality of segments.

[0014] A method of decoding a video according to one embodiment may include a step of decoding substream packets to restore a plurality of segments.

[0015] A method of decoding an image according to one embodiment may include a step of generating a restored image including a plurality of restored segments.

[0016] In one embodiment, the substream packets may correspond to each of the plurality of segments in a predetermined order.

[0017] An encoding device according to one embodiment may include a control unit that identifies a plurality of segments from an image.

[0018] An encoding device according to one embodiment may include at least one core that encodes a plurality of segments in parallel.

[0019] An encoding device according to one embodiment may include a plurality of buffers.

[0020] In one embodiment, the control unit can store substreams generated through parallel encoding in multiple buffers.

[0021] In one embodiment, the control unit can obtain substream packets having a predetermined size from the plurality of buffers when a current state corresponds to a preset output condition while the substreams are stored in the plurality of buffers.

[0022] In one embodiment, the control unit can arrange the substream packets in a predetermined order and output a bitstream in which the substream packets are arranged.

[0023] A decoding device according to one embodiment may include a control unit that identifies a plurality of segments within an image and obtains substream packets of a predetermined size arranged in a predetermined order from a bitstream.

[0024] In one embodiment, substream packets may be generated through parallel encoding of multiple segments.

[0025] A decryption device according to one embodiment may include at least one core that decrypts substream packets to restore a plurality of segments.

[0026] In one embodiment, the substream packets may correspond to each of the plurality of segments in a predetermined order.

[0027] In one embodiment, the control unit can generate a restored image comprising a plurality of restored segments.

[0028] FIG. 1 is a diagram for explaining an image encoding and decoding process according to one embodiment.

[0029] FIG. 2 is a block diagram illustrating the configuration of an encoding device according to one embodiment.

[0030] FIG. 3a is a diagram illustrating images to which parallel encoding can be applied according to one embodiment.

[0031] FIG. 3b is a diagram illustrating slices to which parallel encoding can be applied according to one embodiment.

[0032] FIG. 3c is a diagram illustrating tiles to which parallel encoding can be applied according to one embodiment.

[0033] FIG. 3d is a diagram illustrating a maximum encoding unit row to which parallel encoding can be applied according to one embodiment.

[0034] FIG. 4 is a table for explaining a method of allocating multiple segments according to one embodiment.

[0035] FIG. 5 is a diagram for explaining a process in which multiple segments are encoded in parallel according to one embodiment.

[0036] FIG. 6 is a diagram for explaining a process in which multiple segments are encoded in parallel according to one embodiment.

[0037] FIG. 7a is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.

[0038] FIG. 7b is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.

[0039] FIG. 8 is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.

[0040] FIG. 9 is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.

[0041] Fig. 10 is a flowchart for explaining a method of encoding an image according to one embodiment.

[0042] Fig. 11 is a block diagram illustrating the configuration of a decryption device according to one embodiment.

[0043] FIG. 12 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.

[0044] FIG. 13 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.

[0045] FIG. 14 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.

[0046] Fig. 15 is a flowchart for explaining a method for decoding an image according to one embodiment.

[0047] Fig. 16 is a block diagram illustrating a configuration of a terminal device including an encoding device or a decoding device according to one embodiment.

[0048] A method of encoding an image according to one embodiment may include a step of identifying a plurality of segments from the image.

[0049] A method of encoding an image according to one embodiment may include a step of encoding a plurality of segments in parallel.

[0050] A method for encoding an image according to one embodiment may include a step of storing substreams generated through parallel encoding in a plurality of buffers.

[0051] A method of encoding a video according to one embodiment may include a step of obtaining substream packets having a predetermined size from a plurality of buffers when a current state corresponds to a preset output condition while the substreams are stored in a plurality of buffers.

[0052] A method for encoding an image according to one embodiment may include the steps of arranging substream packets in a predetermined order and outputting a bitstream in which the substream packets are arranged.

[0053] The present disclosure may be subject to various modifications and may have various embodiments. Therefore, embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the embodiments of the present disclosure, and the present disclosure may include all modifications, equivalents, and alternatives that fall within the spirit and technical scope of the various embodiments.

[0054] When describing embodiments, detailed descriptions of related known technologies may be omitted if they are deemed to unnecessarily obscure the gist of the present disclosure. Furthermore, numbers (e.g., "first," "second," etc.) used in the description of embodiments may correspond to identification symbols used to distinguish one component from another.

[0055] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0056] In the present disclosure, when a component is referred to as being “connected” or “connected” to another component, the component may be directly connected or connected to the other component, but unless there is a specific description to the contrary, the component may also be connected or connected via another component in between.

[0057] In the present disclosure, components expressed as "units", "modules", etc. may be two or more components combined into a single component, or a single component may be divided into two or more more detailed components. In addition, each component described below may additionally perform some or all of the functions performed by other components in addition to its own main function, and some of the main functions performed by each component may be exclusively performed by other components.

[0058] In the present disclosure, 'image' may refer to a picture, a still image, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0059] In the present disclosure, a "sample" may refer to data assigned to a sampling location in an image and thus to be processed. For example, a pixel within an image in a spatial domain may correspond to a sample.

[0060] In the present disclosure, a 'block' may include a slice, tile, maximum coding unit, encoding unit, transformation unit, or prediction unit segmented from an image. A block may be composed of one or more samples.

[0061] FIG. 1 is a diagram for explaining an image encoding and decoding process according to one embodiment.

[0062] The encoding unit (110) of the image encoding and decoding system (100) transmits a bitstream generated through encoding of an image, and the decoding unit (150) can output a restored image by decoding the bitstream.

[0063] In one embodiment, the encoding unit (110) may include an encoding device (200) described below, and the decoding unit (150) may include a decoding device (1100) described below.

[0064] In the encoding unit (110), the prediction encoding unit (115) can output prediction data through inter prediction or intra prediction, and the transformation and quantization unit (120) can output quantized transformation coefficients of residual data between the prediction data and the current input image.

[0065] The entropy encoding unit (125) can encode the quantized transform coefficients and output them as a bitstream.

[0066] The quantized transform coefficients are restored as residual data in the spatial domain through the inverse quantization and inverse transformation unit (130), and image data in the spatial domain can be restored from the residual data and the prediction data. The restored image data in the spatial domain can be output as a restored image through the deblocking filtering unit (135) and the loop filtering unit (140). The restored image can be used as a reference image for the next input image through the prediction encoding unit (115).

[0067] Encoded image data among the bitstreams received by the decoding unit (150) can be restored to residual data in the spatial domain through the entropy decoding unit (155) and the inverse quantization and inverse transformation unit (160).

[0068] The prediction data and residual data output from the prediction decoding unit (175) are combined to form image data in the spatial domain, and the deblocking filtering unit (165) and the loop filtering unit (170) can perform filtering on the image data in the spatial domain to output a restored image corresponding to the original image. The restored image can be used as a reference image for the next image by the prediction decoding unit (175).

[0069] The loop filtering unit (140) of the encoding unit (110) can perform loop filtering using filter information input according to user input or system settings.

[0070] The filter information used by the loop filtering unit (140) can be transmitted to the decoding unit (150) together with the encoded image data through the entropy encoding unit (125). The loop filtering unit (170) of the decoding unit (150) can perform loop filtering based on the filter information transmitted to the decoding unit (150).

[0071] Parallel encoding and decoding of images can reduce the time required for encoding and decoding images. Below, an encoding device that encodes images using parallel processing technology is described.

[0072] FIG. 2 is a block diagram illustrating the configuration of an encoding device according to one embodiment.

[0073] The encoding device (200) may include a control unit (210), at least one core (230), and a plurality of buffers (250).

[0074] Although FIG. 2 illustrates that the encoding device (200) includes one core (230), the number of cores (230) that can be included in the encoding device (200) may be multiple.

[0075] In addition, FIG. 2 illustrates that three buffers (250a, 250b, 250c) are included in the encoding device (200), and the number of buffers included in the encoding device (200) may be the same as the number of segments processed in parallel, which will be described later.

[0076] In one embodiment, the control unit (210) and at least one core (230) may be included in a processor.

[0077] In one embodiment, the control unit (210) may be included in the first processor, and at least one core (230) may be included in the second processor.

[0078] In one embodiment, when the encoding device (200) includes a plurality of cores (230), some of the plurality of cores (230) may be included in one processor, and other some of the cores (230) may be included in another processor.

[0079] In one embodiment, the processor for implementing the control unit (210) and at least one core (230) may be a dedicated processor for encoding images. In one embodiment, the processor for implementing the control unit (210) and at least one core (230) may be a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU).

[0080] In one embodiment, at least one instruction for encoding an image may be stored in a memory, and the control unit (210) and at least one core (230) may operate according to the at least one instruction stored in the memory.

[0081] In one embodiment, the plurality of buffers (250) may be physically distinct from one another. In one embodiment, each of the plurality of buffers (250) may represent a storage area identified by hardware or software within one or more buffers.

[0082] In one embodiment, the plurality of buffers (250) may be FIFO (First In First Out) buffers. Accordingly, the data input first into the plurality of buffers (250) may be output first.

[0083] In one embodiment, the control unit (210) can control the operation of at least one core (230) and a plurality of buffers (250) for parallel encoding of an image.

[0084] In one embodiment, the control unit (210) can identify multiple segments from an image for parallel encoding of the image.

[0085] Parallel encoding of multiple segments may imply that the multiple segments are encoded independently. In other words, if there is little dependence between the segments, encoding of one segment may proceed simultaneously with encoding of another segment.

[0086] In one embodiment, there may be some time delay between encoding of one segment and encoding of another segment, such as in wavefront parallel processing (WPP) described below.

[0087] A segment can refer to a unit of an image that is the subject of parallel encoding. For example, a segment can be a block divided from an image. Furthermore, a segment can also be a frame or image included in a video sequence.

[0088] Each of the plurality of segments is assigned to at least one core (230) and can be encoded in parallel by at least one core (230).

[0089] The segments that are the target of parallel encoding are described with reference to FIGS. 3a to 3d.

[0090] FIG. 3a is a diagram illustrating images to which parallel encoding can be applied according to one embodiment.

[0091] In video codecs, groups of pictures (GOPs) can be encoded and decoded according to a hierarchical structure. In this case, the images constituting a GOP are divided into layers, and images in lower layers can be encoded and decoded by referencing images in higher layers.

[0092] If the images in a specific layer do not reference each other, the images in the specific layer can be subject to parallel processing.

[0093] Referring to FIG. 3A, images belonging to layer i+1 can be encoded and decoded with reference to images belonging to layer i. Since images belonging to layer i+1 do not reference each other, the control unit (210) can identify images belonging to the same layer among images included in the image sequence, for example, image 0, image 1, image 2, image 3, image 4, and image 5 illustrated in FIG. 3A, as multiple segments.

[0094] At least one core (230) can independently encode each of image 0, image 1, image 2, image 3, image 4, and image 5. In one embodiment, at least one core (230) can divide each of image 0, image 1, image 2, image 3, image 4, and image 5 into a plurality of maximum coding units, and encode the plurality of maximum coding units according to a predetermined scan order (e.g., a raster scan order).

[0095] FIG. 3b is a diagram illustrating slices to which parallel encoding can be applied according to one embodiment.

[0096] The control unit (210) can divide the image (300) into a plurality of slices. Each of the plurality of slices can include one or more maximum coding units.

[0097] In one embodiment, since the slices do not have dependencies on each other, the control unit (210) can identify slice 0, slice 1, slice 2, and slice 3 as shown in FIG. 3b as multiple segments.

[0098] At least one core (230) can independently encode each of slice 0, slice 1, slice 2, and slice 3. In one embodiment, at least one core (230) can encode the maximum encoding units included in slice 0, slice 1, slice 2, and slice 3 according to a predetermined scan order.

[0099] FIG. 3c is a diagram illustrating tiles to which parallel encoding can be applied according to one embodiment.

[0100] The control unit (210) can divide the image (300) into a plurality of tiles. Each of the plurality of tiles can include one or more maximum encoding units.

[0101] In one embodiment, since the tiles do not have dependencies on each other, the control unit (210) can identify tile 0, tile 1, tile 2, and tile 3 as shown in FIG. 3c as multiple segments.

[0102] At least one core (230) can independently encode each of tile 0, tile 1, tile 2, and tile 3. In one embodiment, at least one core (230) can encode the maximum encoding units included in tile 0, tile 1, tile 2, and tile 3 according to a predetermined scan order.

[0103] FIG. 3d is a diagram illustrating a maximum encoding unit row to which parallel encoding can be applied according to one embodiment.

[0104] WPP (wavefront parallel processing) technology has been adopted in codec standards such as HEVC. In WPP, maximum coding unit rows divided from an image (300) can be processed in parallel.

[0105] The image (300) can be divided into a plurality of maximum coding units, and the maximum coding unit rows within the image (300) can be identified as a plurality of segments. Accordingly, the maximum coding unit row 0, the maximum coding unit row 1, the maximum coding unit row 2, the maximum coding unit row 3, and the maximum coding unit row 4 illustrated in FIG. 3d can be encoded in parallel by at least one core (230).

[0106] In one embodiment of the WPP, a maximum coding unit included in one maximum coding unit row may reference two maximum coding units included in an upper row, which may cause some delay in the process of encoding the maximum coding unit rows in parallel.

[0107] For example, the maximum coding unit (321) located at the leftmost position in the maximum coding unit row 1 can refer to data generated during the encoding process of the maximum coding unit (311) located at the leftmost position in the maximum coding unit row 0 and the maximum coding unit (312) located to the right of the maximum coding unit (311) located at the leftmost position. Accordingly, after the encoding of the two maximum coding units (311, 312) in the maximum coding unit row 0 is completed, the encoding of the maximum coding unit row 1 can begin. Similarly, after the encoding of the two maximum coding units in the maximum coding unit row 1 is completed, the encoding of the maximum coding unit row 2 can begin.

[0108] Referring again to FIG. 2, in one embodiment, when a plurality of segments are identified from an image, the control unit (210) may assign the plurality of segments to at least one core (230) for parallel encoding of the plurality of segments.

[0109] In one embodiment, the control unit (210) may allocate a plurality of segments to at least one core (230) by taking into account performance information of parallel encoding.

[0110] Performance information for parallel encoding may include at least one of the number of cores (230) capable of parallel processing of segments, or the number of threads capable of parallel processing by at least one core (230). In one embodiment, performance information for parallel encoding may also include the number of segments capable of parallel encoding.

[0111] For example, if the number of segments that can be encoded in parallel is n (n is a natural number greater than 1) and the number of segments identified from an image is m (m is a natural number greater than n), then among the m segments, n segments can be first allocated to at least one core (230) and encoded in parallel, and mn segments can be encoded in parallel by at least one core (230) after encoding of the n segments is completed.

[0112] A method of allocating multiple segments to at least one core (230) by considering performance information of parallel encoding is described with reference to FIG. 4.

[0113] FIG. 4 is a table for explaining a method of allocating multiple segments according to one embodiment.

[0114] Five segments are identified from the image, and it is assumed that the number of cores (230) (or threads) capable of parallel processing is three.

[0115] Since three segments can be encoded in parallel at a time, the control unit (210) can first allocate the first segment, the second segment, and the third segment to the first core (or the first thread), the second core (or the second thread), and the third core (or the third thread), respectively. Then, the control unit (210) can additionally allocate the remaining fourth segment and fifth segment to the first core (or the first thread) and the second core (or the second thread).

[0116] The first core, the second core, and the third core can encode the first segment, the second segment, and the third segment in parallel, respectively, and when the encoding of the first segment and the second segment is completed, the first core and the second core can encode the fourth segment and the fifth segment in parallel, respectively.

[0117] In one embodiment, when hyper threading technology capable of generating and processing multiple threads is employed for at least one core (230), the first segment, the second segment, and the third segment may be allocated to the first thread, the second thread, and the third thread. At least one core (230) may parallel encode the first segment, the second segment, and the third segment corresponding to the first thread, the second thread, and the third thread. When the parallel encoding of the first segment and the second segment is completed, at least one core (230) may parallel encode the fourth segment and the fifth segment additionally allocated to the first thread and the second thread.

[0118] In one embodiment, a thread may mean a unit of work, so it may be understood that the fourth segment and the fifth segment are each allocated to a fourth thread and a fifth thread, and the fourth thread and the fifth thread may be processed by at least one core (230).

[0119] FIG. 5 is a diagram for explaining a process in which multiple segments are encoded in parallel according to one embodiment.

[0120] When five segments are identified from the image and the number of cores (230a, 230b, 230c) that can be used for parallel processing is three, the control unit (210) can assign the first to third segments to the first core (230a) to the third core (230c), respectively. Since the number of segments that can be parallel encoded at one time is three, the control unit (210) can additionally assign the fourth and fifth segments to the first core (230a) and the second core (230b).

[0121] Each of the first core (230a), the second core (230b), and the third core (230c) encodes the first segment, the second segment, and the third segment, and substreams (240a, 240b, 240c) may be generated as encoding results. The substreams (240a, 240b, 240c) may include results of entropy encoding data generated through encoding the first segment, the second segment, and the third segment. The substreams (240a, 240b, 240c) may be composed of bins having values ​​of 0 or 1.

[0122] As the first segment, the second segment, and the third segment are encoded in parallel, a first substream (240a) corresponding to the first segment, a second substream (240b) corresponding to the second segment, and a third substream (240c) corresponding to the third segment can be output from each of the first core (230a), the second core (230b), and the third core (230c).

[0123] Each of the first substream (240a), the second substream (240b), and the third substream (240c) can be stored in multiple buffers (250a, 250b, 250c).

[0124] In one embodiment, the number of buffers (250a, 250b, 250c) included in the encoding device (200) may be equal to a number determined from performance information of parallel encoding, for example, the number of cores (230a, 230b, 230c), the number of threads, or the number of segments that can be encoded in parallel.

[0125] As illustrated in FIG. 5, when three segments can be encoded in parallel at a time, the encoding device (200) may include three buffers, i.e., a first buffer (250a), a second buffer (250b), and a third buffer (250c). The first substream (240a), the second substream (240b), and the third substream (240c) may be stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c), respectively.

[0126] Since the fourth segment and the fifth segment are assigned to the first core (230a) and the second core (230b), when the encoding of the first segment and the second segment is completed, each of the first core (230a) and the second core (230b) can additionally encode the fourth segment and the fifth segment. Then, the fourth substream corresponding to the fourth segment can be stored in the first buffer (250a), and the fifth substream corresponding to the fifth segment can be stored in the second buffer (250b).

[0127] FIG. 6 is a diagram for explaining a process in which multiple segments are encoded in parallel according to one embodiment.

[0128] The control unit (210) can assign the first to third segments to the first thread (235a) to the third thread (235c) respectively when five segments are identified from the image and the number of threads (235a, 235b, 235c) that can be processed in parallel at one time is three. In addition, the control unit (210) can additionally assign the fourth segment and the fifth segment to the first thread (235a) and the second thread (235b).

[0129] In one embodiment, since the first segment is assigned to the first thread (235a) and the second segment is assigned to the second thread (235b), the fourth segment may be assigned to the fourth thread, and the fifth segment may be assigned to the fifth thread.

[0130] The core (230) may encode the first segment, the second segment, and the third segment corresponding to the first thread (235a), the second thread (235b), and the third thread (235c), and substreams (240a, 240b, 240c) may be generated as encoding results. The substreams (240a, 240b, 240c) may include results of entropy encoding data generated through encoding the first segment, the second segment, and the third segment. The substreams (240a, 240b, 240c) may be composed of bins having values ​​of 0 or 1.

[0131] As the first segment, the second segment, and the third segment are encoded in parallel, a first substream (240a) corresponding to the first segment, a second substream (240b) corresponding to the second segment, and a third substream (240c) corresponding to the third segment can be output from the core (230).

[0132] The first substream (240a), the second substream (240b), and the third substream (240c) can be stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c), respectively.

[0133] Since the 4th segment and the 5th segment are assigned to the first thread (235a) (or the 4th thread) and the 2nd thread (235b) (or the 5th thread), when the encoding of the 1st segment and the 2nd segment is completed, the core (230) can encode in parallel the 4th segment and the 5th segment assigned to the 1st thread (235a) (or the 4th thread) and the 2nd thread (235b) (or the 5th thread). Then, the 4th substream corresponding to the 4th segment can be stored in the 1st buffer (250a), and the 5th substream corresponding to the 5th segment can be stored in the 2nd buffer (250b).

[0134] Below, the process of outputting a bitstream from the encoding device (200) is described.

[0135] In one embodiment, when encoding of the first segment, the second segment, the third segment, the fourth segment, and the fifth segment illustrated in FIGS. 5 and 6 is completed, a first substream (240a), a second substream (240b), a third substream (240c), a fourth substream, and a fifth substream corresponding to the first segment, the second segment, the third segment, the fourth segment, and the fifth segment, respectively, may be obtained for output from the buffers (250a, 250b, 250c). Then, the first substream (240a), the second substream (240b), the third substream (240c), the fourth substream, and the fifth substream may be arranged in encoding order and output as a bitstream.

[0136] However, according to the output process of the bitstream as above, delays may occur in the encoding and decoding process of the image. Specifically, the time required until the encoding of the first to fifth segments is completed and the time required to arrange the first to fifth substreams (240a) are consumed in the encoding process, so it may not be suitable for a server-client based real-time image transmission system.

[0137] According to one embodiment, time delay can be minimized when substreams stored in multiple buffers (250a, 250b, 250c) are packetized and transmitted even before encoding of segments subject to parallel encoding is completed. This will be described with reference to FIGS. 7a, 7b, 8, and 9.

[0138] FIG. 7a and FIG. 7b are diagrams for explaining a process of outputting a bitstream from a plurality of buffers according to one embodiment.

[0139] The first buffer (250a) can store a first substream corresponding to the encoding result for the first segment, and the second buffer (250b) can store a second substream corresponding to the encoding result for the second segment. In addition, the third buffer (250c) can store a third substream corresponding to the encoding result for the third segment.

[0140] While the first substream, the second substream, and the third substream are stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c) as parallel encoding progresses, the control unit (210) can output the bitstream (700) through the first buffer (250a), the second buffer (250b), and the third buffer (250c) even before parallel encoding is completed, if the current state corresponds to a preset output condition.

[0141] In one embodiment, the control unit (210) can receive output conditions from a user or obtain pre-stored output conditions.

[0142] In one embodiment, the output condition may be set based on at least one of the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c), the ratio between the capacity of the plurality of buffers (250a, 250b, 250c) and the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c), the elapsed time of parallel encoding, or the progress of encoding for each of the plurality of segments.

[0143] For example, when the total size of substreams stored in multiple buffers (250a, 250b, 250c) is 1 megabyte (MB), an output condition may be set such that a bitstream (700) is output from multiple buffers (250a, 250b, 250c).

[0144] In addition, for example, when the ratio of the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) to the capacity of the plurality of buffers (250a, 250b, 250c) becomes 50%, the output condition can be set so that the bitstream (700) is output from the plurality of buffers (250a, 250b, 250c).

[0145] Additionally, for example, an output condition may be set such that when 10 ms has elapsed since parallel encoding for multiple segments has started, a bitstream (700) is output from multiple buffers (250a, 250b, 250c).

[0146] In addition, for example, when encoding of a predetermined number (e.g., 10) of maximum encoding units included in each of a plurality of segments is completed, an output condition may be set such that a bitstream (700) is output from a plurality of buffers (250a, 250b, 250c).

[0147] In one embodiment, two or more of the output conditions exemplified above may be combined. For example, the output condition may be set so that the bitstream (700) is output at the earlier time between the time when the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) becomes 1 megabyte (MB) and the time when the ratio of the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) to the capacity of the plurality of buffers becomes 50%.

[0148] In one embodiment, output conditions may be set based on the degree of delay required by the user or the service to which the encoding process is applied. For example, if low delay is required, the output conditions may be set so that the bitstream (700) is output from the multiple buffers (250a, 250b, 250c) at short intervals. Alternatively, if low delay is not required, the output conditions may be set so that the bitstream (700) is output from the multiple buffers (250a, 250b, 250c) at relatively long intervals.

[0149] For example, when low delay is required, the output condition may be set so that the bitstream (700) is output when the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) is 1 megabyte (MB), and when low delay is not required, the output condition may be set so that the bitstream (700) is output when the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) is 2 megabytes (MB).

[0150] In addition, for example, when low delay is required, the output condition may be set so that the bitstream (700) is output when the ratio of the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) to the capacity of the plurality of buffers (250a, 250b, 250c) is 50%, and when low delay is not required, the output condition may be set so that the bitstream (700) is output when the ratio of the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) to the capacity of the plurality of buffers (250a, 250b, 250c) is 60%.

[0151] In one embodiment, a user can set a delay level for the encoding process, and the control unit (210) can set an output condition according to the delay level set by the user.

[0152] If the current state corresponds to a preset output condition, the control unit (210) can obtain substream packets (715, 725, 735) having a predetermined size from each of a plurality of buffers (250a, 250b, 250c). Here, the substream packet can be understood as a basic transmission unit of a substream included in a bitstream (700).

[0153] For example, if the predetermined size is 8 kilobytes (KB), 8 kilobyte substreams (710, 720, 730) can be extracted as substream packets (715, 725, 735) from each of the plurality of buffers (250a, 250b, 250c).

[0154] Referring to FIG. 7a, if the current state corresponds to a preset output condition, the control unit (210) can obtain substreams (710, 720, 730) including 14 bins corresponding to a predetermined size among the substreams stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c) as substream packets (715, 725, 735). Accordingly, the control unit (210) can obtain a first substream packet (715) including 14 bins from the first buffer (250a), and a second substream packet (725) including 14 bins from the second buffer (250b). In addition, the control unit (210) can obtain a third substream packet (735) including 14 bins from the third buffer (250c).

[0155] When a plurality of substream packets (715, 725, 735) are obtained from a plurality of buffers (250), the control unit (210) can arrange the plurality of substream packets (715, 725, 735) in a predetermined order and output a bitstream (700) in which the plurality of substream packets (715, 725, 735) are arranged.

[0156] In one embodiment, the predetermined order may correspond to the encoding order of the segments corresponding to each of the plurality of substream packets (715, 725, 735). For example, if the encoding order of the first segment corresponding to the first substream packet (715) is earlier than the encoding order of the second segment corresponding to the second substream packet (725), the first substream packet (715) may be arranged before the second substream packet (725).

[0157] In one embodiment, the control unit (210) may arrange the substream packets (715, 725, 735) in an order previously agreed upon between the encoding device (200) and the decoding device (1100). For example, it may be agreed upon between the encoding device (200) and the decoding device (1100) to arrange the substream packets (715, 725, 735) in the reverse order of the encoding order of the plurality of segments.

[0158] In one embodiment, the control unit (210) can generate a bitstream (700) by arranging the substream packets (715, 725, 735) in a predetermined order and adding information necessary to restore the substream packets (715, 725, 735).

[0159] In one embodiment, information necessary to restore the substream packets (715, 725, 735) may be included in the header of the bitstream (700).

[0160] In one embodiment, a bitstream (700) including substream packets (715, 725, 735) may be transmitted to a decryption device (1100) over a network.

[0161] In one embodiment, a bitstream (700) including substream packets (715, 725, 735) may be stored on a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and the like.

[0162] In one embodiment, the control unit (210) may repeatedly arrange sub-stream packets corresponding to the number of segments to be parallel-encoded in a predetermined order. For example, if the number of segments that can be parallel-encoded at one time is 3, sub-stream packets (715, 725, 735) corresponding to each of the three segments may be arranged in a predetermined order, and subsequent sub-stream packets corresponding to each of the three segments may be arranged in a predetermined order behind the previously arranged sub-stream packets (715, 725, 735).

[0163] Referring to FIG. 7b, when the first sub-stream packet (715), the second sub-stream packet (725), and the third sub-stream packet (735) illustrated in FIG. 7a are acquired from the first buffer (250a), the second buffer (250b), and the third buffer (250c) and arranged in the bitstream (700), the sub-streams corresponding to the first sub-stream packet (715), the second sub-stream packet (725), and the third sub-stream packet (735) in the first buffer (250a), the second buffer (250b), and the third buffer (250c) may be deleted.

[0164] If the current state corresponds to a preset output condition, the control unit (210) may acquire a first substream (750) of a predetermined size, a second substream (760) of a predetermined size, and a third substream (770) of a predetermined size, each stored in a plurality of buffers (250a, 250b, 250c), as a first substream packet (755), a second substream packet (765), and a third substream packet (775), and may arrange the acquired substream packets (755, 765, 775) in a predetermined order to generate a bitstream (700).

[0165] According to one embodiment, even if encoding of multiple segments is in progress, if the current state corresponds to a preset output condition, the control unit (210) generates a bitstream by grouping substreams into packets, so that time delay due to parallel encoding and parallel decoding can be minimized.

[0166] FIG. 8 is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.

[0167] While encoding is in progress for multiple segments and multiple substreams are stored in multiple buffers (250a, 250b, 250c), there may be segments whose encoding progress is slower than that of other segments.

[0168] As described above, while the first substream, the second substream, and the third substream are stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c), if the current state corresponds to a preset output condition, the substreams of a predetermined size stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c) can be obtained as substream packets.

[0169] If there is a segment whose encoding progress is slow, the size of the substream generated through encoding for that segment may be smaller than a predetermined size, and thus, it may be difficult to obtain the substream packet from the buffer storing the substream. For example, as illustrated in FIG. 8, the size of the second substream (820) stored in the second buffer (250b) may be smaller than a predetermined size.

[0170] In this case, in one embodiment, instead of obtaining the second sub-stream packet from the second buffer (250b), the control unit (210) may generate a second sub-stream packet (825) of a predetermined size and having a predetermined value (e.g., 0), and arrange the generated second sub-stream packet (825) within the bitstream (800). At this time, the second sub-stream packet (825) may be understood as a garbage packet.

[0171] Accordingly, as illustrated in FIG. 8, the first substream packet (815) corresponding to the first substream (810) stored in the first buffer (250a), the second substream packet (or garbage packet) (825) generated for the second buffer (250b), and the third substream packet (835) corresponding to the third substream (830) stored in the third buffer (250c) can be arranged in a predetermined order.

[0172] Since the second substream (820) stored in the second buffer (250b) is not used to generate the second substream packet (825), it is not deleted from the second buffer (250b) and can be included in the second substream packet at the next output time.

[0173] In one embodiment, the control unit (210) may prevent the decryption device (1100) from performing decryption of the second substream packet (825) by including information in the bitstream (800) indicating that decryption of the second substream packet (825) included in the bitstream (800) is not required. For example, the control unit (210) may include information in the header of the bitstream (800) indicating that decryption of the second substream packet (825) is not required.

[0174] In one embodiment, a constant bitrate can be maintained by generating a substream packet and including the generated substream packet in the bitstream (800) even if the substream packet cannot be obtained from the buffer.

[0175] FIG. 9 is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.

[0176] As previously described with reference to FIGS. 5 and 6, each of the fourth segment and the fifth segment can be additionally allocated to the first core (230a) (or the first thread (235a)) and the second core (230b) (or the second thread (235b)), so that encoding of the fourth segment and the fifth segment can continue even after encoding of the first segment, the second segment, and the third segment is completed.

[0177] Referring to FIG. 9, when the encoding for the third segment is completed and all of the third substreams stored in the third buffer (250c) are output, no more data may be stored in the third buffer (250c). However, since encoding for the fourth and fifth segments is in progress, the fourth substream generated through encoding for the fourth segment may be stored in the first buffer (250a), and the fifth substream generated through encoding for the fifth segment may be stored in the second buffer (250b).

[0178] If the current state corresponds to a preset output condition, the control unit (210) can obtain a fourth substream (910) of a predetermined size stored in the first buffer (250a) as a fourth substream packet (915), and can obtain a fifth substream (920) of a predetermined size stored in the second buffer (250b) as a fifth substream packet (925). In addition, since no substream is stored in the third buffer (250c), the control unit (210) can generate a sixth substream packet (935) of a predetermined size and having a predetermined value.

[0179] The control unit (210) can arrange the fourth sub-stream packet (915), the fifth sub-stream packet (925), and the sixth sub-stream packet (or garbage packet) (935) in a predetermined order, and generate a bitstream (900) including the fourth sub-stream packet (915), the fifth sub-stream packet (925), and the sixth sub-stream packet (935).

[0180] In one embodiment, the control unit (210) may add information to the bitstream (900) that decryption of the sixth substream packet (935) is not required.

[0181] Fig. 10 is a flowchart for explaining a method of encoding an image according to one embodiment.

[0182] At step S1010, the encoding device (200) can identify multiple segments from the image.

[0183] In one embodiment, each of the plurality of segments may correspond to an image constituting an image sequence, a slice segmented from an image, a tile segmented from an image, or a maximum coding unit row segmented from an image.

[0184] In one embodiment, each of the plurality of segments may include one or more maximum coding units.

[0185] In step S1020, the encoding device (200) can encode multiple segments in parallel.

[0186] In one embodiment, the encoding device (200) may assign a plurality of segments to a plurality of cores (230) and / or a plurality of threads for parallel encoding of the plurality of segments.

[0187] For allocation of multiple segments, the encoding device (200) may consider performance information of parallel encoding.

[0188] In one embodiment, performance information of parallel encoding may include the number of cores (230) capable of parallel processing of segments, the number of threads capable of parallel processing by at least one core (230), or the number of segments capable of parallel encoding.

[0189] In one embodiment, the encoding device (200) can determine the number of segments that can be encoded in parallel at one time, and can assign the identified number of segments among the plurality of segments identified from the image to a plurality of cores (230) and / or a plurality of threads. In addition, if there are remaining segments, the encoding device (200) can additionally assign the remaining segments to a plurality of cores (230) and / or a plurality of threads.

[0190] The encoding device (200) can encode a plurality of segments in parallel using at least one core (230). In one embodiment, the encoding device (200) can encode each of the plurality of segments based on a maximum encoding unit.

[0191] In step S1030, the encoding device (200) can store substreams generated during parallel encoding of multiple segments in multiple buffers (250).

[0192] In one embodiment, a plurality of segments to be encoded in parallel and a plurality of buffers may correspond 1:1. Accordingly, a plurality of substreams generated through encoding of each of the plurality of segments may be stored 1:1 in the plurality of buffers.

[0193] In one embodiment, the number of the plurality of buffers (250) may be equal to the number of cores (230) capable of parallel processing of segments, the number of threads capable of parallel processing by at least one core (230), or the number of segments capable of parallel encoding.

[0194] In step S1040, the encoding device (200) can obtain substream packets having a predetermined size from the plurality of buffers (250) when the current state corresponds to a preset output condition while the substreams are stored in the plurality of buffers (250).

[0195] In one embodiment, the output condition may be set based on at least one of the total size of the substreams stored in the plurality of buffers (250), the ratio between the capacity of the plurality of buffers (250) and the total size of the substreams stored in the plurality of buffers (250), the elapsed time of parallel encoding, or the progress of encoding for each of the plurality of segments.

[0196] In step S1050, the encoding device (200) can arrange the substream packets in a predetermined order and output a bitstream including the substream packets arranged in the predetermined order.

[0197] In one embodiment, the bitstream may further include additional information necessary to reconstruct multiple segments from substream packets. In one embodiment, the additional information may be included in a header of the bitstream.

[0198] In one embodiment, the predetermined order may correspond to the encoding order of the plurality of segments.

[0199] In one embodiment, the predetermined order may correspond to a pre-arranged order between the encoding device (200) and the decoding device (1100).

[0200] In one embodiment, the encoding device (200) may repeatedly arrange substream packets obtained from a plurality of buffers (250) in a predetermined number of sequences according to a predetermined order. Here, the predetermined number may correspond to a number identified from performance information of parallel encoding.

[0201] Hereinafter, a decoding device (1100) corresponding to the above-described encoding device (200) will be described with reference to FIGS. 11 to 14.

[0202] FIG. 11 is a block diagram illustrating the configuration of a decryption device (1100) according to one embodiment.

[0203] The decryption device (1100) may include a control unit (1110) and at least one core (1130).

[0204] Although FIG. 11 illustrates that the decryption device (1100) includes one core (1130), the number of cores (1130) that may be included in the decryption device (1100) may be multiple.

[0205] In one embodiment, the control unit (1110) and at least one core (1130) may be included in a processor.

[0206] In one embodiment, the control unit (1110) may be included in a first processor, and at least one core (1130) may be included in a second processor.

[0207] In one embodiment, when the decryption device (1100) includes multiple cores (1130), some of the multiple cores (1130) may be included in one processor, and other some may be included in another processor.

[0208] In one embodiment, the processor for implementing the control unit (1110) and at least one core (1130) may be a dedicated processor for decoding images. In one embodiment, the processor for implementing the control unit (1110) and at least one core (1130) may be a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU).

[0209] In one embodiment, at least one instruction for decrypting an image may be stored in a memory, and the control unit (1110) and at least one core (1130) may operate according to the at least one instruction stored in the memory.

[0210] The control unit (1110) can obtain a bitstream corresponding to the encoding result of the image.

[0211] In one embodiment, the control unit (1110) can receive a bitstream from the encoding device (200) over a network.

[0212] In one embodiment, the bitstream may be obtained from a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and the like.

[0213] In one embodiment, the bitstream may include substream packets of a predetermined size, arranged in a predetermined order.

[0214] In one embodiment, the control unit (1110) can identify multiple segments within an image to be restored to restore the image from substream packets.

[0215] Each of the plurality of segments may correspond to an image constituting a video sequence, a slice segmented from an image, a tile segmented from an image, or a maximum coding unit row segmented from an image.

[0216] In one embodiment, the control unit (1110) transmits substream packets obtained from the bitstream to at least one core (1130), and the at least one core (1130) can decode the substream packets to restore a plurality of segments.

[0217] Substream packets included in a bitstream can correspond to each of multiple segments in a predetermined order.

[0218] In one embodiment, the predetermined order may correspond to the decoding order of the plurality of segments. Accordingly, if the decoding order of the first segment is earlier than the decoding order of the second segment, the substream packet arranged earlier among the first substream packet and the second substream packet may correspond to the first segment, and the substream packet arranged later may correspond to the second segment.

[0219] In one embodiment, the control unit (1110) may skip decoding of a substream packet if the bitstream contains information indicating that any of the substream packets is not subject to decoding. Accordingly, the substream packet may be deleted, and the next substream packet may be decoded by at least one core (1130).

[0220] The above-described encoding device (200) can encode a plurality of segments in parallel to generate substream packets. In one embodiment, at least one core (1130) can decode substream packets included in a bitstream in parallel, or can sequentially decode substream packets in a predetermined order.

[0221] In one embodiment, if the decoding device (1100) has the same parallel encoding performance as the encoding device (200), the control unit (1110) can allocate substream packets that can be parallel decoded at one time to multiple cores (1130) or multiple threads.

[0222] In one embodiment, the control unit (1110) may predetermine which segment each of the substream packets included in the bitstream corresponds to. As the substream packets are decoded by at least one core (1130), the segments corresponding to the substream packets may be restored.

[0223] Since the control unit (1110) can determine in advance which segment each of the substream packets is for, parallel decoding of the substream packets may not necessarily be necessary. This is because even if the substream packets are decoded sequentially, it is possible to know which segment the decoded data is for.

[0224] In one embodiment, the control unit (1110) can check in advance the performance information of the parallel encoding of the encoding device (200). The performance information of the parallel encoding can be included in the bitstream.

[0225] The control unit (1110) can determine from the performance information of parallel encoding how many substream packets are arranged in a predetermined order, and accordingly determine which segment each substream packet belongs to. This will be described with reference to FIGS. 12 to 14.

[0226] FIG. 12 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.

[0227] Referring to FIG. 12, the bitstream may include a first substream packet to a seventh substream packet.

[0228] In one embodiment, the control unit (1110) can determine how often substream packets are arranged and repeated from the performance information of parallel encoding.

[0229] For example, if the number confirmed from the performance information of parallel encoding is 4, the control unit (1110) can know that the substream packets are arranged in a predetermined order every 4.

[0230] The control unit (1110) can identify a first segment, a second segment, a third segment, and a fourth segment from the image. The control unit (1110) can determine that the first sub-stream packet corresponds to the first segment, the second sub-stream packet corresponds to the second segment, the third sub-stream packet corresponds to the third segment, and the fourth sub-stream packet corresponds to the fourth segment, according to the decoding order of the first segment, the second segment, the third segment, and the fourth segment or an order agreed upon in advance with the encoding device (200).

[0231] If the number confirmed from the performance information is 4, the substream packets are arranged in a predetermined order every 4, so the control unit (1110) can determine that the 5th substream packet corresponds to the 1st segment, the 6th substream packet corresponds to the 2nd segment, and the 7th substream packet corresponds to the 3rd segment.

[0232] At least one core (1130) can decrypt the first substream packet and the fifth substream packet to restore the first segment, decrypt the second substream packet and the sixth substream packet to restore the second segment, and decrypt the third substream packet and the seventh substream packet to restore the third segment. Furthermore, at least one core (1130) can decrypt the fourth substream packet to restore the fourth segment.

[0233] FIG. 13 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.

[0234] In the embodiment illustrated in FIG. 13, it is assumed that the first segment, the second segment, the third segment, and the third segment are identified from the image, and the number of segments confirmed from the performance information is 3.

[0235] Accordingly, the first substream packet and the fourth substream packet may correspond to the first segment, the second substream packet and the fifth substream packet may correspond to the second segment, and the third substream packet and the sixth substream packet may correspond to the third segment.

[0236] In one embodiment, if the restoration of the first segment from the fourth substream packet is not completed, the seventh substream packet may correspond to the first segment. This is because the substream packets are arranged in a predetermined order every three, and the seventh substream packet, which is located at the seventh position, corresponds to the first segment according to the predetermined order.

[0237] In one embodiment, if the restoration of the first segment is completed through decoding of the fourth substream packet, the seventh substream packet located at the seventh position may correspond to the fourth segment. That is, this may mean that the first segment and the fourth segment were sequentially encoded through one core (1130) or one thread.

[0238] FIG. 14 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.

[0239] In one embodiment, the control unit (1110) may obtain information from the bitstream that a second substream packet among the substream packets in the bitstream is not a target for decoding. In this case, decoding of the second substream packet may be skipped.

[0240] If the number confirmed from the performance information of parallel encoding is 4 and the first to fourth segments are identified from the video, the control unit (1110) may determine that the first substream packet corresponds to the first segment. The control unit (1110) may determine that the second substream packet does not correspond to any segment because it is not a target of decoding.

[0241] The control unit (1110) can determine that the third sub-stream packet corresponds to the third segment, and the fourth sub-stream packet corresponds to the fourth segment. Since the sub-stream packets are arranged in a repeating manner every four, the control unit (1110) can determine that the fifth sub-stream packet corresponds to the first segment, the sixth sub-stream packet corresponds to the second segment, and the seventh sub-stream packet corresponds to the third segment.

[0242] At least one core (1130) can decrypt the first substream packet and the fifth substream packet to restore the first segment, decrypt the sixth substream packet to restore the second segment, and decrypt the third substream packet and the seventh substream packet to restore the third segment. Furthermore, at least one core (1130) can decrypt the fourth substream packet to restore the fourth segment.

[0243] In one embodiment, the control unit (1110) can generate a restored image using the restored plurality of segments when the plurality of segments are restored by at least one core (1130).

[0244] In one embodiment, the control unit (1110) can combine a plurality of restored segments to generate a restored image.

[0245] In one embodiment, filtering (e.g., deblocking filtering) may be applied to the boundaries between restored segments.

[0246] In one embodiment, when a restored image is generated, the control unit (1110) can transmit the restored image to a playback device such as a display.

[0247] Fig. 15 is a flowchart for explaining a method for decoding an image according to one embodiment.

[0248] At step S1510, the decryption device (1100) can identify multiple segments within the image.

[0249] In one embodiment, each of the plurality of segments may correspond to an image constituting an image sequence, a slice segmented from an image, a tile segmented from an image, or a maximum coding unit row segmented from an image.

[0250] In one embodiment, each of the plurality of segments may include one or more maximum coding units.

[0251] At step S1520, the decryption device (1100) can obtain substream packets of a predetermined size arranged in a predetermined order from the bitstream.

[0252] In one embodiment, the decoding device (1100) can obtain performance information of parallel encoding of the encoding device (200) from the bitstream, and based on the obtained performance information, can determine how often the substream packets are arranged and repeated.

[0253] At step S1530, the decryption device (1100) can decrypt substream packets through at least one core (1130) to restore multiple segments.

[0254] In one embodiment, the decryption device (1100) can decrypt the substream packets in parallel by assigning each of the number of substream packets identified from the performance information to a plurality of cores (1130) or a plurality of threads.

[0255] In one embodiment, the substream packets may correspond to each of the plurality of segments in a predetermined order, so that at least one core (1130) may restore the plurality of segments using the substream packets corresponding to each of the plurality of segments.

[0256] In one embodiment, the decryption device (1100) can restore each of the plurality of segments based on a maximum coding unit.

[0257] At step S1540, the decryption device (1100) can generate a restored image using the restored plurality of segments.

[0258] In one embodiment, the decryption device (1100) can combine a plurality of restored segments to generate a restored image.

[0259] In one embodiment, filtering (e.g., deblocking filtering) may be applied to the boundaries between restored segments.

[0260] In one embodiment, when a restored image is generated, the decryption device (1100) can transmit the restored image to a playback device such as a display.

[0261] FIG. 16 is a block diagram illustrating a configuration of a terminal device including an encoding device (200) or a decoding device (1100) according to one embodiment.

[0262] Referring to FIG. 16, a terminal device (1600) according to one embodiment may include a tuner unit (1640), a processor (1610), a display unit (1620), a communication unit (1650), a detection unit (1630), an input / output unit (1670), a video processing unit (1680), an audio processing unit (1685), an audio output unit (1660), a memory (1690), and a power supply unit (1695).

[0263] A tuner unit (1640) according to one embodiment can select and tune only the frequency of a channel to be received among many radio wave components through amplification, mixing, resonance, etc. of a broadcast signal received wired or wirelessly. The broadcast signal includes audio, video, and additional information (e.g., EPG (Electronic Program Guide)).

[0264] The tuner unit (1640) can receive broadcast signals from various sources, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner unit (1640) can also receive broadcast signals from sources, such as analog broadcasting or digital broadcasting.

[0265] The detection unit (1630) detects the user's voice, the user's image, or the user's interaction, and may include a microphone (1631), a camera unit (1632), and a light receiving unit (1633).

[0266] The microphone (1631) receives the user's spoken voice. The microphone (1631) can convert the received voice into an electrical signal and output it to the processor (1610). The user's voice may include, for example, a voice corresponding to a menu or function of the terminal device (1600).

[0267] The camera unit (1632) can receive images (e.g., consecutive frames).

[0268] The optical receiver (1633) receives an optical signal (including a control signal) from an external control device through an optical window (not shown) of a bezel of the display unit (1620). The optical receiver (1633) can receive an optical signal corresponding to a user input (e.g., touch, pressing, touch gesture, voice, or motion) from the control device. A control signal can be extracted from the received optical signal under the control of the processor (1610).

[0269] The input / output unit (1670) receives video (e.g., moving images, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from the outside of the terminal device (1600) under the control of the processor (1610). The input / output interface may include any one of HDMI (High-Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), DVI (Digital Visual Interface), component jack, and PC port.

[0270] The processor (1610) controls the overall operation of the terminal device (1600) and the signal flow between the internal components of the terminal device (1600), and performs the function of processing data. When there is a user input or a preset stored condition is satisfied, the processor (1610) can execute the OS (Operating System) stored in the memory (1690) and various applications.

[0271] The processor (1610) may include a RAM that stores signals or data input from the outside of the terminal device (1600) or is used as a storage area corresponding to various tasks performed in the terminal device (1600) (e.g., image noise removal task), a ROM that stores a control program for controlling the terminal device (1600), and a processor.

[0272] The video processing unit (1680) performs processing on video data received by the terminal device (1600). The video processing unit (1680) can perform various image processing such as encoding, decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on the video data.

[0273] In one embodiment, the encoding process of the aforementioned encoding device (200) or the decoding process of the decoding device (1100) may be performed by the video processing unit (1680).

[0274] The audio processing unit (1685) processes audio data. The audio processing unit (1685) may perform various processing operations, such as decoding, amplification, and noise filtering, on audio data. Meanwhile, the audio processing unit (1685) may include multiple audio processing modules to process audio corresponding to multiple contents.

[0275] The audio output unit (1660) outputs audio included in a broadcast signal received through the tuner unit (1640) under the control of the processor (1610). The audio output unit (1660) can output audio (e.g., voice, sound) input through the communication unit (1650) or the input / output unit (1670). In addition, the audio output unit (1660) can output audio stored in the memory (1690) under the control of the processor (1610). The audio output unit (1660) can include at least one of a speaker, a headphone output terminal, or an S / PDIF (Sony / Philips Digital Interface:) output terminal.

[0276] The power supply unit (1695) supplies power input from an external power source to components inside the terminal device (1600) under the control of the processor (1610). In addition, the power supply unit (1695) can supply power output from one or more batteries (not shown) located inside the terminal device (1600) to the internal components under the control of the processor (1610).

[0277] The memory (1690) can store various data, programs or applications for driving and controlling the terminal device (1600) under the control of the processor (1610). The memory (1690) can include a broadcast reception module, a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, an optical reception module, a display control module, an audio control module, an external input control module, a power control module, a power control module for an external device connected wirelessly (e.g., Bluetooth), a voice database (DB), or a motion database (DB). The modules and database of the memory (1690) can be implemented in the form of software to perform a mirroring function, a broadcast reception control function, a channel control function, a volume control function, a communication control function, a voice recognition function, a motion recognition function, an optical reception control function, a display control function, an audio control function, an external input control function, a power control function or a power control function for an external device connected wirelessly (e.g., Bluetooth) in the terminal device (1600). The processor (1610) can perform each function using the software stored in the memory (1690).

[0278] Meanwhile, the block diagram of the terminal device (1600) illustrated in FIG. 16 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted depending on the specifications of the terminal device (1600) actually implemented. That is, two or more components may be combined into one component, or one component may be subdivided into two or more components, as needed. In addition, the functions performed by each block are for the purpose of explaining embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

[0279] One embodiment aims to minimize the delay between the process of encoding an image and the process of decoding the image.

[0280] Additionally, one embodiment aims to minimize delay that may occur during the output process of a bitstream generated through parallel encoding of an image.

[0281] Additionally, one embodiment aims to provide a codec suitable for applications requiring low latency characteristics, such as wireless TV, cloud gaming, AR (Augmented Reality) and VR (Virtual Reality).

[0282] The technical problems to be achieved through the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure pertains from the description below.

[0283] A method of encoding an image by an encoding device (200) according to one embodiment may include a step of identifying a plurality of segments from the image.

[0284] A method of encoding an image by an encoding device (200) according to one embodiment may include a step of encoding a plurality of segments in parallel.

[0285] A method of encoding an image by an encoding device (200) according to one embodiment may include a step of storing substreams generated through parallel encoding in a plurality of buffers (250).

[0286] A method of encoding an image by an encoding device (200) according to one embodiment may include a step of obtaining substream packets having a predetermined size from a plurality of buffers (250) when a current state corresponds to a preset output condition while the substreams are stored in a plurality of buffers (250).

[0287] A method of encoding an image by an encoding device (200) according to one embodiment may include a step of arranging substream packets in a predetermined order and outputting a bitstream in which the substream packets are arranged.

[0288] According to a method for encoding an image according to one embodiment, a time delay that may occur during the process of encoding an image can be minimized.

[0289] In one embodiment, the preset output condition may be set based on at least one of the total size of the substreams stored in the plurality of buffers (250), the ratio between the capacity of the plurality of buffers (250) and the total size of the substreams stored in the plurality of buffers (250), the elapsed time of parallel encoding, or the progress of encoding for each of the plurality of segments.

[0290] According to one embodiment, various output conditions can be set, so that the amount of delay occurring during the image encoding process can be adjusted in various ways.

[0291] In one embodiment, the predetermined order corresponds to an encoding order of a plurality of segments, and if the encoding order of the first segment is earlier than the encoding order of the second segment, the substream packet corresponding to the first segment may be arranged before the substream packet corresponding to the second segment.

[0292] In one embodiment, since the substream packets are arranged in a predetermined order, it is easy to determine which segment each substream packet corresponds to.

[0293] In one embodiment, the outputting step may include, when the size of a substream stored in a first buffer among the plurality of buffers (250) is less than a predetermined size, generating a substream packet of a predetermined size, the substream packet being composed of bits of a predetermined value; and arranging the generated substream packets in a predetermined order.

[0294] In one embodiment, a constant bitrate can be maintained by generating garbage packets even if there is not enough data stored in the buffer.

[0295] In one embodiment, the step of outputting the bitstream may include the step of including information in the bitstream that the generated substream packet is not subject to decryption.

[0296] In one embodiment, decoding of certain substream packets may be skipped, thereby preventing segments from being incorrectly restored.

[0297] In one embodiment, a plurality of segments may be encoded in parallel by a plurality of cores (230a, 230b, 230c), or a plurality of segments may be assigned to a plurality of threads (235a, 235b, 235c), and the plurality of threads (235a, 235b, 235c) may be encoded in parallel by at least one core (230).

[0298] According to one embodiment, rapid encoding of images may be possible through various types of parallel encoding.

[0299] In one embodiment, each of the plurality of segments may include a slice, a tile, or a maximum coding unit row.

[0300] According to one embodiment, parallel encoding is performed based on slices, tiles, or maximum encoding unit rows specified in a video standard, so that it can be utilized in various applications.

[0301] A method of decoding an image by a decoding device (1100) according to one embodiment may include a step of identifying a plurality of segments within the image.

[0302] A method for decoding an image using a decoding device (1100) according to one embodiment may include a step of obtaining substream packets of a predetermined size arranged in a predetermined order from a bitstream. In one embodiment, the substream packets may be generated through parallel encoding of a plurality of segments.

[0303] A method of decoding an image by a decoding device (1100) according to one embodiment may include a step of decoding substream packets to restore a plurality of segments.

[0304] A method of decoding an image by a decoding device (1100) according to one embodiment may include a step of generating a restored image including a plurality of restored segments.

[0305] In one embodiment, the substream packets may correspond to each of the plurality of segments in a predetermined order.

[0306] According to a method for decoding an image according to one embodiment, time delay that may occur during an image encoding process and an image decoding process can be minimized.

[0307] In one embodiment, if the bitstream includes information that a first substream packet among the substream packets is not a target for decoding, decoding of the first substream packet may be skipped.

[0308] In one embodiment, decoding of certain substream packets may be skipped, thereby preventing segments from being incorrectly restored.

[0309] In one embodiment, the method for decoding an image further includes the step of obtaining information on the number of cores or threads used for parallel encoding from a bitstream, and the substream packets may be arranged in a predetermined order for each number of cores or threads.

[0310] In one embodiment, since the substream packets are arranged in a predetermined order, it is easy to determine which segment each substream packet corresponds to.

[0311] In one embodiment, when the number of cores or threads is n (where n is a natural number greater than 1) and the number of multiple segments is m (where m is a natural number greater than or equal to n), n substream packets are arranged in a predetermined order, and each of the n substream packets can correspond to each of n segments among the m segments in the predetermined order.

[0312] An encoding device (200) according to one embodiment may include a control unit (210) that identifies a plurality of segments from an image.

[0313] An encoding device (200) according to one embodiment may include at least one core (230) that encodes a plurality of segments in parallel.

[0314] An encoding device (200) according to one embodiment may include a plurality of buffers (250).

[0315] In one embodiment, the control unit (210) can store substreams generated through parallel encoding in a plurality of buffers (250).

[0316] In one embodiment, the control unit (210) may obtain substream packets having a predetermined size from the plurality of buffers (250) when the current state corresponds to a preset output condition while the substreams are stored in the plurality of buffers (250).

[0317] In one embodiment, the control unit (210) can arrange substream packets in a predetermined order and output a bitstream in which the substream packets are arranged.

[0318] According to an encoding device (200) according to one embodiment, a time delay that may occur during the encoding process of an image can be minimized.

[0319] In one embodiment, the control unit (210) may, when the size of a substream stored in a first buffer among the plurality of buffers (250) is less than a predetermined size, generate a substream packet of a predetermined size, consisting of bits of a predetermined value, and arrange the generated substream packets in a predetermined order.

[0320] In one embodiment, a constant bitrate can be maintained by generating garbage packets even if there is not enough data stored in the buffer.

[0321] A decoding device (1100) according to one embodiment may include a control unit (1110) that identifies a plurality of segments within an image and obtains substream packets of a predetermined size arranged in a predetermined order from a bitstream.

[0322] A decryption device (1100) according to one embodiment may include at least one core (1130) that decrypts substream packets to restore a plurality of segments.

[0323] In one embodiment, substream packets are generated through parallel encoding of a plurality of segments, and may correspond to each of the plurality of segments in a predetermined order.

[0324] In one embodiment, the control unit (1110) can generate a restored image including a plurality of restored segments.

[0325] According to a decoding device (1100) according to one embodiment, time delay that may occur during an image encoding process and an image decoding process can be minimized.

[0326] One embodiment can minimize the delay between the process of encoding an image and the process of decoding the image.

[0327] Additionally, one embodiment can minimize delay that may occur during the output process of a bitstream generated through parallel encoding of an image.

[0328] Additionally, one embodiment may provide a codec suitable for applications requiring low latency characteristics, such as wireless TV, cloud gaming, AR (Augmented Reality), and VR (Virtual Reality).

[0329] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0330] Meanwhile, the embodiments of the present disclosure described above can be written as a program that can be executed on a computer, and the written program can be stored in a storage medium that can be read by a device.

[0331] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0332] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0333] Above, the technical idea of ​​the present disclosure has been described in detail with reference to preferred embodiments, but the technical idea of ​​the present disclosure is not limited to the above embodiments, and various modifications and changes are possible by a person having ordinary knowledge in the art within the scope of the technical idea of ​​the present disclosure.

Claims

1. In a method of encoding an image using an encoding device (200), A step of identifying multiple segments from an image; A step of parallel encoding the above plurality of segments; A step of storing substreams generated through the above parallel encoding in multiple buffers (250); While the above substreams are stored in the plurality of buffers (250), if the current state corresponds to a preset output condition, a step of obtaining substream packets having a predetermined size from the plurality of buffers (250); and A method for encoding a video, comprising the steps of arranging the substream packets in a predetermined order and outputting a bitstream in which the substream packets are arranged.

2. In paragraph 1, The above preset output conditions are: A method for encoding a video, wherein the method is set based on at least one of the total size of the substreams stored in the plurality of buffers (250), the ratio between the capacity of the plurality of buffers (250) and the total size of the substreams stored in the plurality of buffers (250), the elapsed time of the parallel encoding, or the progress of encoding for each of the plurality of segments.

3. In any one of paragraphs 1 and 2, The above predetermined order is, Corresponding to the encoding order of the above multiple segments, A method for encoding a video, wherein if the encoding order of the first segment is earlier than the encoding order of the second segment, the substream packet corresponding to the first segment is arranged earlier than the substream packet corresponding to the second segment.

4. In any one of the clauses 1 to 3, The above output step is, When the size of the substream stored in the first buffer among the plurality of buffers (250) is less than the predetermined size, a step of generating a substream packet of the predetermined size, which is composed of bits of a predetermined value; and A method for encoding a video, comprising the step of arranging the generated substream packets in the predetermined order.

5. In any one of the clauses 1 to 4, The step of outputting the above bitstream is: A method for encoding a video, comprising the step of including information in the bitstream that the generated substream packet is not a target of decoding.

6. In any one of paragraphs 1 to 5, A method for encoding a video, wherein the plurality of segments are encoded in parallel by a plurality of cores (230a, 230b, 230c), or the plurality of segments are allocated to a plurality of threads (235a, 235b, 235c), and the plurality of threads (235a, 235b, 235c) are encoded in parallel by at least one core (230).

7. In any one of paragraphs 1 to 6, Each of the above multiple segments, A method of encoding an image, comprising slices, tiles or rows of maximum coding units.

8. A computer-readable recording medium having recorded thereon a program for performing on a computer the method of encoding an image according to any one of claims 1 to 7.

9. In a method of decoding an image using a decoding device (1100), A step of identifying multiple segments within an image; A step of obtaining substream packets of a predetermined size arranged in a predetermined order from a bitstream, the substream packets being generated through parallel encoding of the plurality of segments; a step of decrypting the above substream packets to restore the plurality of segments; and A step of generating a restored image including the restored plurality of segments, A method for decoding a video, wherein the substream packets correspond to each of the plurality of segments in the predetermined order.

10. In paragraph 9, A method for decoding a video, wherein if the bitstream includes information that a first substream packet among the substream packets is not a target of decoding, decoding of the first substream packet is skipped.

11. In any one of the clauses 9 to 10, The method of decrypting the above video is as follows: Further comprising a step of obtaining information on the number of cores or threads used for the parallel encoding from the bitstream, A method for decoding a video, wherein the substream packets are arranged in a predetermined order for each number of cores or threads.

12. In any one of paragraphs 9 to 11, A method for decoding a video, wherein when the number of the cores or threads is n (n is a natural number greater than 1) and the number of the plurality of segments is m (m is a natural number greater than or equal to n), n substream packets are arranged in the predetermined order, and each of the n substream packets corresponds to each of n segments among the m segments in the predetermined order.

13. A control unit (210) for identifying multiple segments from an image; At least one core (230) for parallel encoding of the plurality of segments; and Including multiple buffers (250), The above control unit (210) The substreams generated through the above parallel encoding are stored in the plurality of buffers (250), While the above substreams are stored in the plurality of buffers (250), if the current state corresponds to a preset output condition, substream packets having a predetermined size are obtained from the plurality of buffers (250), An encoding device that arranges the above substream packets in a predetermined order and outputs a bitstream in which the above substream packets are arranged.

14. In paragraph 13, The above control unit (210) An encoding device that generates a substream packet of the predetermined size, which is composed of bits of a predetermined value, when the size of a substream stored in a first buffer among the plurality of buffers (250) is less than the predetermined size, and arranges the generated substream packets in the predetermined order.

15. A control unit (1110) that identifies multiple segments in an image and obtains substream packets of a predetermined size arranged in a predetermined order from a bitstream; and At least one core (1130) for decrypting the above substream packets and restoring the plurality of segments, The above substream packets are generated through parallel encoding of the above multiple segments, The above substream packets correspond to each of the above plurality of segments in the above predetermined order, The above control unit (1110) is a decoding device that generates a restored image including the restored plurality of segments.

Images