Method, device, and recording medium for encoding / decoding images for machines

Abstract

A bitstream used by a method, apparatus, and recording medium for encoding / decoding an image based on a region of interest of the present disclosure comprises at least one Network Abstraction Layer (NAL) unit, and the header of the NAL unit may include index information specifying the type of the NAL unit.

Description

Technology Field

[0001] The present disclosure relates to a method, apparatus, and recording medium for encoding / decoding images for a machine. Background Technology

[0002] Traditionally, video encoding and decoding technologies have achieved improved compression efficiency and image quality by considering the human visual system. However, in the future, video encoding and decoding technologies are expected to be widely utilized not only for human vision but also in machine vision fields such as surveillance, intelligent transportation, smart cities, and intelligent factories.

[0003] Accordingly, there is a need to develop image encoding / decoding technology capable of achieving high-efficiency compression and recognition accuracy by simultaneously considering both human vision and machine vision. The problem to be solved

[0004] The present disclosure may aim to vary the NAL packet generation process by Group of Frames (GOP) and frame by frame depending on the scenario. means of solving the problem

[0005] A bitstream used by a method, apparatus, and recording medium for encoding / decoding an image based on a region of interest of the present disclosure comprises at least one Network Abstraction Layer (NAL) unit, and the header of the NAL unit may include index information specifying the type of the NAL unit. Effects of the invention

[0006] The present disclosure can improve video encoding / decoding efficiency by varying the NAL packet generation process by GOP (Group Of Frames) and frame by frame depending on the scenario. Brief explanation of the drawing

[0007] FIG. 1 is a block diagram of an image encoder according to one embodiment of the present disclosure. FIG. 2 is a block diagram of an image decoder according to one embodiment of the present disclosure. Figure 3 illustrates a first embodiment of the VCM Process. FIG. 4 illustrates an example of a pre-defined table of VCM NAL unit types. Figure 5 illustrates an example of General VCM NAL unit syntax and VCM NAL unit header syntax. FIG. 6 illustrates a second embodiment of the VCM Process of the present disclosure. FIG. 7 illustrates an example of a VCM service scenario of the present disclosure. Figure 8 illustrates a specific example of an initialization packet. Figure 9 illustrates a specific example where a single VCM process is applied. Figure 10 illustrates a specific example of a case where complex VCM processes are applied. Figure 11 illustrates an example of a GOP and a frame. Figure 12 illustrates the process of generating a NAL packet for the first I1 frame of the first GOP of the entire sequence. Figure 13 illustrates the process of generating NAL packets for the B1 and B2 frames of the first GOP of the entire sequence. Figure 14 illustrates the process of generating a NAL packet for the P1 frame of the first GOP of the entire sequence. Figure 15 illustrates the process of generating NAL packets for frames B3 and B4 of the first GOP of the entire sequence. Figure 16 illustrates the process of generating a NAL packet for the first I2 frame of the second GOP of the entire sequence. Figure 17 illustrates the process of generating NAL packets for the B5 and B6 frames of the second GOP of the entire sequence. Figure 18 illustrates the process of generating a NAL packet for the P2 frame of the second GOP of the entire sequence. Figure 19 illustrates the process of generating NAL packets for the B7 and B8 frames of the second GOP of the entire sequence. FIG. 20 illustrates an example of VCM NAL data. FIG. 21 illustrates an example of a pre-defined table of the VCM NAL type. Specific details for implementing the invention

[0008] The present disclosure is subject to various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. Similar reference numerals in the drawings refer to the same or similar functions across various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. The detailed description of exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments is limited only by the appended claims, together with all equivalents to those claimed therein, provided they are properly described.

[0009] In this disclosure, terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0010] Where it is stated that any component of the present disclosure is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, or that there may be other components in between. On the other hand, where it is stated that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0011] The components shown in the embodiments of the present disclosure are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the rights of the present disclosure as long as they do not depart from the essence of the present disclosure.

[0012] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this disclosure, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, the description in this disclosure that a specific configuration "comprising" does not exclude configurations other than that configuration, but means that additional configurations may be included within the scope of the practice or technical concept of this disclosure.

[0013] Some components of the present disclosure may not be essential components performing an essential function in the present disclosure, but may be optional components merely for enhancing performance. The present disclosure may be implemented by including only the components essential to embody the essence of the present disclosure, excluding components used merely for enhancing performance, and a structure including only the essential components, excluding optional components used merely for enhancing performance, is also included within the scope of the rights of the present disclosure.

[0014] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions may obscure the gist of this specification, such detailed description is omitted, and the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0015] In the present disclosure, unlike the existing VCM structure, functionality can be extended by defining a class of a VCM-dedicated NAL unit type.

[0016] In addition, the present disclosure discloses an improved bitstream of VCM, an NAL sample stream structure, and an NAL unit type class structure.

[0017] In addition, the present disclosure optimizes the size of the VCM bitstream by separating the SRD and PRD parameters within the RSD sample stream into NAL formats for time, space, ROI, and bit depth, and by removing and reconstructing redundant parameters.

[0018] FIG. 1 is a block diagram of an image encoder according to one embodiment of the present disclosure.

[0019] Referring to FIG. 1, the image encoder may include a preprocessing unit (110) and an image encoding unit (120).

[0020] The preprocessing unit (110) performs a preprocessing process to convert the input original images into images suitable for image encoding. At this time, the images input to the preprocessing unit (110) may be color or black-and-white images following the YUV or YCbCr format.

[0021] The preprocessing unit (110) may include at least one of a temporal resampling unit (112), a spatial resampling unit (114), or a region of interest-based processing unit (116).

[0022] The temporal resampling unit (112) resamples the images temporally. Only the resampled images can be selected as targets for image encoding. That is, through temporal resampling, encoding for some of the images input to the preprocessing unit (110) can be omitted. For example, odd-numbered images of a 60fps (frames per second) image can be omitted to convert the 60fps image into a 30fps image. Alternatively, images of a specific output order can be omitted by considering the temporal overlap between the images.

[0023] The spatial resampling unit (114) spatially resamples the image. Through spatial resampling, the size and / or spatial resolution of the image may be reduced. For example, an image with a resolution of 1920x1080 can be converted into an image of 960x540 or 480x270, etc.

[0024] The region of interest-based processing unit (116) sets a region of interest within the image so that image encoding / decoding is performed primarily on information important for machine inference tasks. The region of interest-based processing unit (116) can remove background areas excluding the set region of interest, or adjust the size and / or position of the region of interest within the image so that the region of interest is encoded / decoded with high quality.

[0025] The video encoding unit (120) can encode the video output from the preprocessing unit (110). Meanwhile, the video encoding unit (120) can encode the video by utilizing a conventional codec technology or a codec technology modified for VCM (Video Coding for Machine) based on a conventional codec technology. For example, the video encoding unit (120) can encode the video based on HEVC, VVC, or AV1. A bitstream is generated as a result of the video encoding, and the generated bitstream can be transmitted to a video decoder.

[0026] FIG. 2 is a block diagram of an image decoder according to one embodiment of the present disclosure.

[0027] Referring to FIG. 2, the image decoder may include an image decoder (210) and a post-processing unit (220).

[0028] The image decoding unit (210) decodes the bitstream received from the image encoder (110) to generate a decoded (or reconstructed) image. The image decoding unit (210) can decode the bitstream based on the codec technology used in the image encoding unit (120).

[0029] The post-processing unit (220) performs post-processing on the decoded image. Through post-processing, the size and frame rate of the image can be restored to match the original image.

[0030] The post-processing unit (220) may include at least one of a post-filtering unit (222), a region of interest-based restoration unit (224), a spatial restoration unit (226), or a temporal restoration unit (218).

[0031] The post-filtering unit (222) applies filtering to reduce the restoration error of the decoded image. For example, the post-filtering unit (222) may apply an in-loop filter to the decoded image. The in-loop filter may include at least one of a deblocking filter, a sample adaptive offset filter, an LMCS (Luma mapping chroma scaling), or an adaptive loop filter.

[0032] In the region of interest-based restoration unit (224), an image of the same size as the original image is obtained based on the region of interest information. For example, if a cropped image containing the region of interest is encoded, the decoded image will have a different size from the original image. Accordingly, the region of interest-based restoration unit can adjust the retargeted image to the original size. Here, the retargeted image may represent the decoded image or an image that has been upscaled through the spatial restoration unit (226). Alternatively, if the size or location of the region of interest within the image to be encoded is adjusted, the region of interest-based restoration unit (224) can adjust the location and size of the region of interest within the retargeted image to match the original image.

[0033] In the spatial restoration unit (226), upscaling is performed on the decoded image. Through upscaling, the decoded image can be restored to the same size and / or spatial resolution as the original image.

[0034] In the temporal restoration unit (228), an image of a temporal location where encoding / decoding was omitted is restored by temporal resampling. Specifically, the temporal restoration unit (228) can generate an image of a temporal location where encoding / decoding was omitted through interpolation between decoded images.

[0035] Meanwhile, additional information may be encoded and signaled to perform reverse processing on the image processing performed in the preprocessing unit (110). In the postprocessing unit (220), postprocessing of the decoded image may be performed based on the additional information to generate an image for machine inference to be performed on a machine. Meanwhile, the additional information may be referred to as 'metadata'.

[0036] Metadata may include at least one of temporal resampling information, spatial resampling information, or region of interest processing information.

[0037] Temporal resampling information may include at least one of a flag indicating whether temporal resampling has been performed or information indicating a temporal resampling rate.

[0038] For example, the above flag being 1 indicates that temporal resampling has been performed. In this case, information indicating the temporal resampling rate may be additionally encoded / decoded. When temporal resampling is performed, fewer images than the number of original images may be encoded / decoded. In the image decoder, images that have been omitted from encoding / decoding can be restored through temporal restoration.

[0039] On the other hand, the above flag being 0 indicates that temporal resampling was not performed.

[0040] The temporal resampling rate can be expressed as a power of 2. For example, a temporal resampling rate of 2^N indicates that one of 2^N images is selected as the image to be encoded / decoded. For instance, only images with a POC (Picture Order Count) that is a multiple of 2^N can be the targets for encoding / decoding. Information representing the temporal resampling rate can represent the power of the temporal resampling rate (i.e., N). For example, the above information can represent the value of the power of the temporal resampling rate or the value obtained by subtracting 1 from the power of the rate.

[0041] Spatial resampling information may include at least one of a flag indicating whether spatial resampling has been performed, or information indicating a scaling parameter for spatial resampling.

[0042] For example, the above flag being 1 indicates that spatial resampling has been performed. In this case, information representing scaling parameters may be additionally encoded. Specifically, information representing horizontal scaling parameters and information representing vertical scaling parameters may each be encoded and signaled. When spatial resampling is performed, the size and / or spatial resolution of the image may be reduced. In the image decoder, the size of the decoded image can be restored to the size of the original image or a preset size through spatial reconstruction. Meanwhile, information for specifying the preset size may be additionally encoded / decoded.

[0043] The above flag being 0 indicates that spatial resampling was not performed.

[0044] The region of interest processing information may include at least one of image size information or region of interest information.

[0045] The image size information may include information indicating whether retargeting has been performed. A retargeting flag of 1 indicates that the retargeted image is encoded / decoded instead of the original image. On the other hand, a retargeting flag of 0 indicates that the original image is encoded / decoded as is.

[0046] A retargeting image represents an image generated by performing at least one of resolution adjustment and position adjustment on at least one region of interest within an original image. Accordingly, the resolution or position of the region of interest within the retargeted image may differ from that of the original image. Additionally, the size of the retargeting image may be the same as or smaller than that of the original image.

[0047] If retargeting is allowed (i.e., when the retargeting flag is 1), size information of the retargeted image can be encoded / decoded. The size information of the retargeted image may include width information of the image and height information of the image.

[0048] Meanwhile, additional encoding / decoding information indicating the size difference between the original image and the retargeted image may be performed. For example, information indicating whether the difference in size between the retargeted image and the original image is encoded / decoded may be encoded / decoded.

[0049] For example, information indicating whether the magnitude difference is encoded / decoded is 0, which indicates that the magnitude difference between the retargeted image and the original image is not encoded / decoded.

[0050] On the other hand, if the information indicating whether the size difference is encoded / decoded is 1, it indicates that the size difference between the retargeted image and the original image is encoded / decoded. In this case, information indicating the difference in size between the retargeted image and the original image may be additionally encoded / decoded.

[0051] Information indicating the size difference represents the size difference between the original image and the retargeted image. Meanwhile, information indicating the size difference in the horizontal direction and information indicating the size difference in the horizontal direction can each be encoded and signaled.

[0052] The region of interest information may include at least one of a flag indicating whether a region of interest exists, number information of the region of interest, scaling parameters of the region of interest, or location information of the region of interest.

[0053] For example, the above flag being 1 indicates that information about the region of interest is encoded / decoded. In this case, at least one of the number of regions of interest, scaling parameter information of the region of interest, location information of the region of interest, or size information of the region of interest may be additionally encoded / decoded.

[0054] On the other hand, if the above flag is 0, it indicates that there is no region of interest.

[0055] Information on the number of regions of interest indicates the number of regions of interest. Meanwhile, the number of regions of interest can be calculated in units of image groups containing at least one image.

[0056] The scaling parameter of the region of interest represents the scaling parameter for the region of interest. Depending on the scaling parameter of the region of interest, the size of the region of interest can be adjusted.

[0057] The scaling parameter information of the region of interest may include information indicating whether the scaling parameter of the region of interest has been updated. If the information indicating whether to update indicates that the scaling parameter of the region of interest has not been updated, the scaling parameter of the region of interest may be set to a default value or the same value as in the previous frame. On the other hand, if the information indicating whether to update indicates that the scaling parameter of the region of interest should be updated, the information indicating the scaling parameter of the region of interest may be additionally encoded / decoded.

[0058] Meanwhile, scaling parameter information for the region of interest can be encoded / decoded individually for each region of interest.

[0059] The location information of the region of interest indicates the location of the region of interest in the original image. At this time, the horizontal position information (i.e., x-axis coordinate) and the vertical position information (i.e., y-axis coordinate) of the region of interest can be encoded / decoded, respectively.

[0060] The size information of the region of interest represents the size of the region of interest in the original image. In this case, the horizontal size (i.e., width) and vertical size (i.e., height) information of the region of interest can be encoded and decoded, respectively.

[0061] As described above, according to the present disclosure, through an image preprocessing process, the encoding / decoding efficiency of the image can be improved while maintaining machine mission performance.

[0062] RBSP of the present disclosure may be an abbreviation for Raw Byte Sequence Payload.

[0063] VCM of the present disclosure may be an abbreviation for Video Coding for Machines.

[0064] NAL in the present disclosure may be an abbreviation for Network Abstraction Layer.

[0065] NUT of the present disclosure may be an abbreviation for NAL Unit Type.

[0067] Figure 3 illustrates a first embodiment of the VCM Process.

[0068] The VCM encoder encodes each result value of the sequentially executed Processing Tools into NAL data, and the VCM decoder can decode and utilize each NAL data encoded as the input values ​​of the sequentially executed Reconstruction Tools.

[0069] FIG. 4 illustrates an example of a pre-defined table of VCM NAL unit types.

[0070] The ICVD (Initialization Codec Video Data) NAL unit can be transmitted as needed when the first IDR frame of the entire sequence enters the VCM encoder.

[0071] The SPS (Sequence Parameter Set) NAL unit may be transmitted for each frame as needed. Examples of the sequence parameter set NAL unit are VSPS_NUT, T1_SPS_NUT, T2_SPS_NUT, T3_SPS_NUT, ..., T n_ It may include SPS_NUT.

[0072] The PPS (Picture Parameter Set) NAL unit can be transmitted for each frame as needed. An example of a Picture Parameter Set NAL unit is VPPS_NUT, T x It may include _PPS_NUT and VVD_NUT.

[0073] The following may be included as VSPS (VCM Video Sequence Parameter Set) parameters.

[0074] - vcmsps_id

[0075] - vcmsps_log2_max_restoration_data_picture_order_cnt_lsb_minus4

[0076] - profile_tier_level

[0077] - ptl_profile_codec_group_idc

[0078] VSPS parameters can be transmitted periodically or non-periodically as needed.

[0079] The ICVD (Initialization video codec information) parameters may include the following.

[0080] - vps (codec video parameter set)

[0081] ICVD parameters can be transmitted periodically or non-periodically as needed.

[0082] One embodiment of the ICVD (Initialization video codec information) parameter may include the following.

[0083] 1) VPS(video parameter set)

[0084] (1) Codec identification and profile info

[0085] - Codec identifier: Identifier of the codec used (e.g., avc, hevc, vvc)

[0086] - Profile idc: Profile information used for encoding

[0087] - Tier flag: Profile tier information

[0088] - Level idc: Level information

[0089] - Profile compatibility flag: Profile compatibility information

[0090] (2) Time constraint

[0091] - num_units_in_tick: Time unit for frame rate calculation

[0092] - time_scale: number of time units per second (fps = time_scale / num_units_in_tick)

[0093] - CVDClockTick: Decoding timing information (num_units_in_tick / time_scale)

[0094] 2) SPS(sequence parameter set)

[0095] (1) Resolution and color info

[0096] - Width, height: Video width, height

[0097] - Chroma format idc: Color format (ex: 4:2:0, 4:2:2, 4:4:4)

[0098] - Bit depth luma, bit depth chroma: luma, chroma bit depth

[0099] (2) Frame reference and order information

[0100] - log2_max_pic_order_cnt_lsb_minus4: Calculate Picture Order Count (POC)

[0101] 3) PPS(picture parameter set)

[0102] (1) Frame reference and order information

[0103] - num_ref_frame: Number of available frames

[0104] In addition, additional information required by the inner codec at the VPS / SPS / PPS level may be included.

[0105] Figure 5 illustrates an example of General VCM NAL unit syntax and VCM NAL unit header syntax.

[0107] FIG. 6 illustrates a second embodiment of the VCM Process of the present disclosure.

[0108] The VCM bitstream structure is designed to be similar to the MPEG V3C (Versatile Video Coding-based 3D Visual Content Coding) bitstream structure and can provide flexibility and scalability through a hierarchical structure. The VCM bitstream of the second embodiment of the present disclosure may have a structure as shown in FIG. 6.

[0109] Each NAL data can be divided into a NAL header and a NAL payload. The NAL header provides the type and identification information of the NAL unit, and the NAL payload may contain recovery data or parameter information according to the type of the NAL unit.

[0110] Each NAL Unit is divided into VCL (Video Coding Layer) and non-VCL (Non-Video Coding Layer); VCL refers to a layer containing actual image pixel data and compressed image data, while non-VCL refers to a layer providing metadata and parameter information for decoding video data.

[0111] Each step of FIG. 6 (e.g., temporal resampling) can be performed in each unit. For example, temporal resampling can be performed in a Temporal Resampling Unit. That is, each block of FIG. 6 can represent a step or a unit of a device.

[0112] Referring to FIG. 6, the VCM encoder may include a temporal resampling unit, a spatial resampling unit, an ROI-based processing unit, a bit depth reduction unit, an internal encoding unit, and a VCM NAL packing unit. The VCM decoder may include a VCM NAL parsing unit, an internal decoding unit, an ROI-based processing unit, a spatial resampling unit, a temporal resampling unit, and a bit depth reduction unit.

[0113] In the VCM encoder, NAL data corresponding to the output value of each unit can be obtained from each unit of the VCM encoder, and the NAL data can be packed by the VCM NAL packing section and transmitted to the VCM decoder.

[0114] In the VCM decoder, the packed NAL data received from the encoder in the VCL NAL parsing unit is decoded, and the decoded NAL data can be utilized in each unit of the VCM decoder corresponding to the VCM encoder.

[0115] FIG. 7 illustrates an example of a VCM service scenario of the present disclosure.

[0116] The initialization packet may be a structure designed to allow the decoder to receive essential initialization information related to the video codec before the video data is transmitted in earnest. Figure 7(A) shows the structure of the initialization packet, and internal codec initialization data may be transmitted through a NAL Unit of type ICVD_NUT. Through the initialization packet, the decoder can be prepared to decode the video data to be transmitted thereafter.

[0117] When a single VCM process is applied, it represents a scenario in which a bitstream is configured by performing only a specific single process (temporal resampling, spatial resampling, region of interest-based processing, bit depth reduction processing, etc.). Figure 7(B) illustrates an example where only temporal resampling is performed. In this case, the bitstream can be composed of an ICVD_NUT initialization packet and TSPS (Temporal Sequence Parameter Set)_NUT and TPPS (Temporal Picture Parameter Set)_NUT for temporal resampling. This scenario can operate very efficiently in a simple VCM environment and can maximize transmission efficiency by configuring the bitstream with a minimal set of parameters.

[0118] When a complex VCM process is applied, it may represent a scenario in which a bitstream is constructed by performing two or more processes (temporal resampling, spatial resampling, region of interest-based processing, bit depth reduction processing, etc.). Figure 7(C) illustrates an example in which the entire process is applied. In this case, the bitstream may consist of sequence and picture-level NALs corresponding to all processes, starting from the initialization packet (ICVD_NUT), and relevant parameters for the internal codec. Through this, the decoder can efficiently restore the compressed video data after temporal resampling processing.

[0119] Figure 8 illustrates a specific example of an initialization packet.

[0120] Figure 9 illustrates a specific example where a single VCM process is applied.

[0121] Figure 10 illustrates a specific example of a case where complex VCM processes are applied.

[0122] A VCM service scenario can be selected by the encoder, and the bitstream structure transmitted to the decoder can be determined accordingly. For example, the encoder can determine the VCM service scenario for each frame or frame group by considering the location of the GOP within the entire sequence and the location of the frame within the GOP.

[0123] The decoder can explicitly determine the VCM service scenario through information about the VCM service scenario, or implicitly determine the VCM service scenario through the structure of the transmitted VCM service scenario. Additionally, the VCM service scenario may be pre-defined in the decoding / encoding unit.

[0124] It can be confirmed that the proposed VCM bitstream structure demonstrates efficiency compared to existing V3C-based structures in terms of structural simplicity, improved parsing efficiency due to layer reduction, and reduced transmission overhead resulting from the elimination of redundant headers. For example, in a single-process scenario, transmission capacity can be reduced by including only the minimum essential NAL Units; even in a complex-process scenario, since NAL Units corresponding to each process are managed independently, the decoder can selectively parse only the necessary processes.

[0125] Figure 11 illustrates an example of a GOP and a frame.

[0126] The VCM scenario can be determined based on the position of the target GOP (Group of Pictures) and the position of the target picture (frame) within the entire sequence.

[0127] For example, the VCM scenario is determined by considering factors such as whether it is the first GOP in the entire sequence or the first frame within the GOP, and the NAL packing structure may differ depending on the determined VCM scenario.

[0128] Referring to FIG. 11, for example, the entire sequence is composed of two GOPs, and the same picture parameters are shared between frames I1, B1, and B2, the same picture parameters are shared between frames P1, B3, and B4, and the same picture parameters are shared between frames I2, B5, B6, P2, B7, and B8.

[0129] Figure 12 illustrates the process of generating a NAL packet for the first I1 frame of the first GOP of the entire sequence.

[0130] For the I1 frame, which is the first frame of the first GOP of the entire sequence, the NAL packet generation process can be performed as follows.

[0131] 1) Can generate ICVD NAL data.

[0132] (ICVD NAL data is generated only in the first frame of the first GOP of the entire sequence, and may not be generated in the first frame of other GOPs.)

[0133] 2) Sequence information NAL data can be generated.

[0134] 3) Can generate NAL data for pictures and picture information.

[0135] 4) The generated NAL data can be packed.

[0136] Figure 13 illustrates the process of generating NAL packets for the B1 and B2 frames of the first GOP of the entire sequence.

[0137] For frames B1 and B2 of the first GOP in the entire sequence, ICVD NAL data may not be generated because they are not the first frames of the first GOP. Additionally, since they share picture parameters with frame I1, sequence information NAL data generation may not be performed. In other words, the NAL packet generation process for frames B1 and B2 may be as follows.

[0138] 1) Can generate NAL data for pictures and picture information.

[0139] 2) The generated NAL data can be packed.

[0140] Figure 14 illustrates the process of generating a NAL packet for the P1 frame of the first GOP of the entire sequence.

[0141] In the case of the P1 frame of the first GOP of the entire sequence, ICVD NAL data may not be generated because it is not the first frame of the first GOP. Additionally, although it shares picture parameters with the I1 frame, it may be newly generated as the temporal picture parameter set is different. In other words, the NAL packet generation process for the P1 frame may be as follows.

[0142] 1) NAL data can be generated for the picture and picture information (including a set of temporal picture parameters).

[0143] 2) The generated NAL data can be packed.

[0144] Figure 15 illustrates the process of generating NAL packets for frames B3 and B4 of the first GOP of the entire sequence.

[0145] For frames B3 and B4 of the first GOP in the entire sequence, ICVD NAL data may not be generated because they are not the first frames of the first GOP. Additionally, picture parameters are shared with frame I1, and temporal picture parameters may be shared with frame P1.

[0146] In other words, the process of generating NAL packets for frames B3 and B4 can be as follows.

[0147] 1) Can generate NAL data for pictures and picture information.

[0148] 2) The generated NAL data can be packed.

[0149] Figure 16 illustrates the process of generating a NAL packet for the first I2 frame of the second GOP of the entire sequence.

[0150] In the case of the I2 frame, which is the first frame of the second GOP in the entire sequence, ICVD NAL data may not be generated because it is not the first frame of the first GOP. Additionally, since it has separate picture parameters from the first GOP, the NAL packet generation process for the I2 frame may be as follows.

[0151] 1) Sequence information NAL data can be generated.

[0152] 2) Can generate NAL data for pictures and picture information.

[0153] 3) The generated NAL data can be packed.

[0154] Figure 17 illustrates the process of generating NAL packets for the B5 and B6 frames of the second GOP of the entire sequence.

[0155] For frames B5 and B6 of the second GOP in the entire sequence, ICVD NAL data may not be generated because they are not the first frame of the first GOP. Additionally, since they share picture parameters with the I2 frame, the generation of sequence information NAL data may not be performed. In other words, the NAL packet generation process for frames B5 and B6 may be as follows.

[0156] 1) Can generate NAL data for pictures and picture information.

[0157] 2) The generated NAL data can be packed.

[0158] Figure 18 illustrates the process of generating a NAL packet for the P2 frame of the second GOP of the entire sequence.

[0159] In the case of the P2 frame of the second GOP in the entire sequence, ICVD NAL data may not be generated because it is not the first frame of the first GOP. Additionally, since it shares picture parameters with the I2 frame, the generation of sequence information NAL data may not be performed. Furthermore, unlike P1, it does not have separate temporal picture parameters, so a separate set of temporal picture parameters may not be generated. In other words, the NAL packet generation process for the P2 frame may be as follows.

[0160] 1) Can generate NAL data for pictures and picture information.

[0161] 2) The generated NAL data can be packed.

[0162] Figure 19 illustrates the process of generating NAL packets for the B7 and B8 frames of the second GOP of the entire sequence.

[0163] For frames B7 and B8 of the second GOP in the entire sequence, ICVD NAL data may not be generated because they are not the first frame of the first GOP. Additionally, sequence information NAL data generation may not be performed because they share picture parameters with the I2 frame. In other words, the NAL packet generation process for frames B7 and B8 may be as follows.

[0164] 1) Can generate NAL data for pictures and picture information.

[0165] 2) The generated NAL data can be packed.

[0166] FIG. 20 illustrates an example of VCM NAL data.

[0167] FIG. 21 illustrates an example of a pre-defined table of the VCM NAL type.

[0168] ICVD(Initialization Codec Video Data)_NUT may be a NAL Unit containing initial configuration information for the operation of internal codecs (HEVC, VVC, etc.) prior to the start of the VCM decoding process. For example, it can serve the role of transmitting parameter values ​​required by the codec at the VCM level in advance, such as codec profile, resolution, and frame rate.

[0169] VVD(VCM Video Data)_NUT is a VCL NAL Unit that transmits actual compressed video data within a VCM bitstream and may contain video data encoded through an internal codec. The payload of the NAL Unit may include NAL Units defined by the internal codec.

[0170] EOB(End Of Bitstream)_NUT is a NAL Unit that specifies the end of a VCM bitstream; the decoder determines whether to end parsing through this NAL Unit, and can either ignore residual data or safely terminate the stream.

[0171] VSPS_NUT / VPPS_NUT may include common settings for video preprocessing or specific frame parameters.

[0172] TSPS_NUT / TPPS_NUT may include parameters required for the temporal resampling process.

[0173] SSPS_NUT / SPPS_NUT may include parameters required for the spatial resampling process.

[0174] RSPS_NUT / RPPS_NUT may include parameters related to region of interest processing.

[0175] BSPS_NUT / BPPS_NUT may include parameters required for bit depth reduction processing.

[0177] The names of the syntax elements introduced in the embodiments described above are merely temporary for the purpose of describing the embodiments according to the present disclosure. Syntax elements may be named with names different from those proposed in the present disclosure.

[0178] The components described in the exemplary embodiments of the present disclosure may be implemented by hardware elements. For example, the hardware elements may include at least one of a digital signal processor (DSP), a processor, a controller, an application-specific integrated circuit (ASIC), a programmable logic element such as an FPGA, a GPU, other electronic devices, or a combination thereof. At least some of the functions or processes described in the exemplary embodiments of the present disclosure may be implemented in software, and the software may be recorded on a recording medium. The components, functions, and processes described in the exemplary embodiments may be implemented by a combination of hardware and software.

[0179] A method according to one embodiment of the present disclosure may be implemented as a program that can be executed by a computer, and said computer program may be recorded on various recording media such as magnetic storage media, optical reading media, digital storage media, etc.

[0180] The various technologies described in this disclosure may be implemented as digital electronic circuits or computer hardware, firmware, software, or a combination thereof. The technologies may be implemented as computer program products, namely, computer programs tangibly implemented on information media or computer programs (e.g., machine-readable storage devices (e.g., computer-readable media) or data processing devices), or as computer programs implemented as signals processed by or propagated to perform operations of data processing devices (e.g., programmable processors, computers, or a plurality of computers).

[0181] Computer program(s) may be written in any form of programming language, including compiled or interpreted languages, and may be distributed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs may be executed on a single computer, or by multiple computers distributed across one site or multiple sites and interconnected by a communication network.

[0182] Examples of processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and one or more processors of a digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both. Components of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Additionally, the computer may include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or may be connected to said mass storage devices to receive and / or transmit data. Examples of information media suitable for implementing computer program instructions and data include semiconductor memory devices (magnetic media such as hard disks, floppy disks, and magnetic tapes), optical media such as compact disc read-only memory (CD-ROM) and digital video discs (DVD), magneto-optical media such as floptical disks, and Read Only Memory (ROM), Random Access Memory (RAM), flash memory, Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), and other known computer-readable media. Processors and memory may be complemented or integrated by special-purpose logic circuits.

[0183] A processor may execute an operating system (OS) and one or more software applications running on the OS. A processor unit may also access, store, manipulate, process, and generate data in response to software execution. For simplification, a processor unit is described in the singular; however, those skilled in the art will understand that the processor unit may include multiple processing elements and / or various types of processing elements. For example, a processor unit may include multiple processors or a processor and a controller. It may also constitute different processing structures, such as parallel processors. Furthermore, a computer-readable medium means any medium accessible to a computer and may include both computer storage media and transmission media.

[0184] The present disclosure includes detailed descriptions of various detailed embodiments, but such details are not intended to limit the invention or claims proposed in the present disclosure and should be understood as describing the features of specific exemplary embodiments.

[0185] Features individually described in exemplary embodiments in this disclosure may be implemented by a single exemplary embodiment. Conversely, various features described with respect to a single exemplary embodiment in this disclosure may be implemented by a combination of multiple exemplary embodiments or a suitable sub-combination. Furthermore, in this disclosure, said features may operate by a specific combination and may be described as said combination first claimed, but in some cases, one or more features may be excluded from the claimed combination, or the claimed combination may be changed into a sub-combination or a modified form of a sub-combination.

[0186] Likewise, even if operations are described in a specific order in the drawings, it should not be understood that it is necessary to execute the operations in a specific sequence or order, or that all operations must be performed, in order to obtain the desired result. In certain cases, multitasking and parallel processing may be useful. Furthermore, it should not be understood that the various device components in the exemplary embodiments of all embodiments must be separated, and the aforementioned program components and devices may be packaged into a single software product or multiple software products.

[0187] The exemplary embodiments disclosed in this specification are merely illustrative and are not intended to limit the scope of this disclosure. Those skilled in the art will recognize that various modifications to the exemplary embodiments may be made without departing from the spirit and scope of the claims and their equivalents.

[0188] Accordingly, the present disclosure shall be deemed to include all other substitutions, modifications, and changes falling within the scope of the following claims.

Claims

Claim 1 A video encoding method comprising: a step of encoding an output video through a preprocessing process to generate a bitstream; and a step of transmitting the bitstream, wherein the bitstream comprises at least one Network Abstraction Layer (NAL) unit, and the header of the NAL unit comprises index information specifying the type of the NAL unit.

Images