Optical correction SEI message for video coding

SEI messages are used to encode optical correction parameters, addressing camera-induced distortions like barrel or pincushion distortion, chromatic aberration, and vignetting, ensuring high-quality video rendering by compensating for these distortions.

KR1020260164579APending Publication Date: 2026-11-17TENCENT AMERICA LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
KR1020267032854
Authority / Receiving Office
KR · KR
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-04-07
Filing Date
2025-04-08
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing video encoding and decoding technologies lack effective implementations of optical correction parameters in Supplementary Enhancement Information (SEI) messages, leading to distorted video output due to camera lens distortions such as barrel or pincushion distortion, chromatic aberration, and vignetting, which are not adequately addressed in the context of SEI messages.

Method used

Incorporation of SEI messages to encode optical correction parameters, including lens optical correction (loc) SEI messages, to apply corrections for radial distortion, chromatic aberration, and vignetting, with metadata indicating focal centers and lens models, enabling accurate video decoding and rendering.

Benefits of technology

Enables accurate optical correction of distorted video content, ensuring high-quality video rendering by compensating for camera-induced distortions, thereby improving video quality and compatibility across various devices and networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure PCT00079_ABST
    Figure PCT00079_ABST
Patent Text Reader

Abstract

A video encoding and decoding method based on a supplementary enhancement information (SEI) message representing a picture of a video and at least one parameter of an optical correction to be applied to at least one picture associated with the picture.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 631,899 filed April 9, 2024 and U.S. Application No. 19 / 171,805 filed April 7, 2025, the entire contents of which are expressly incorporated into this application by reference.

[0003] The subject of the present disclosure relates to video coding and decoding, and more specifically to the syntax and semantics of Supplementary Enhancement Information (SEI) messages that encode optical correction parameters. Background Technology

[0004] Video coding and decoding using inter-picture prediction with motion compensation have been known for decades. Uncompressed digital video can consist of a series of pictures, each of which may have spatial dimensions of, for example, 1920×1080 luminance samples and associated chrominance samples. These series of pictures may have a fixed or variable picture rate (informally also called the frame rate), such as, for example, 60 pictures per second or 60 Hz. Uncompressed video has significant bit rate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a 60 Hz frame rate) requires bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 GBytes of storage space.

[0005] The purpose of video coding and decoding may be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned requirements for bandwidth or storage space, and in some cases, reduce them by more than double orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to a technique where an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to use the reconstructed signal for its intended purpose. For video, lossy compression is widely used. The degree of acceptable distortion varies depending on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television contribution applications. The achievable compression ratio may reflect the fact that higher acceptable / tolerable distortion yields a higher compression ratio.

[0006] Video encoders and decoders can utilize a wide range of techniques, including, for example, motion compensation, transformation, quantization, and entropy coding, some of which are described below.

[0007] Optical correction of source or reconstructed video, including some models mentioned below, has been known for decades. An image or video transmission chain may include rendering devices such as cameras, networks, decoders, and screens coupled to an encoder. As described below, a camera may include lenses and / or other optical elements capable of distorting optical input from a straight representation that a screen sometimes expects in a specific way. In some cases, for example in cost-effective action cameras, these cameras do not include compensation techniques to form a true straight representation of the captured scene. Without compensation for artifacts introduced by optical elements in the camera or renderer, rendered pictures may appear distorted. Since each camera may have different distortions, parameters must be established to instruct the renderer to compensate for distortions introduced by optical elements within the camera and passed from the encoder to the renderer through the transmission chain and decoder. Certain distortions from linear representations may also be introduced by techniques outside the camera, for example, for artistic purposes, and such distortions may also need to be compensated for in the renderer.

[0008] Common forms of distortion include barrel or pincushion distortion, which commonly occurs when using fisheye or wide-angle lenses in cameras; chromatic aberration, resulting from different diffraction of light of different wavelengths / colors in the lens; and vignetting (non-linear light transmission characteristics). Furthermore, while optical correction of source or reconstructed video, including some models mentioned below, has been known for decades, there have been no specific applications or particularly useful implementations in the context of SEI messages.

[0009] As such, technical challenges exist in the context of video encoding and decoding where SEI messages may be particularly useful, and the embodiments of this specification provide solutions to these challenges.

[0010] The apparatus and method include a memory configured to store computer program code and a processor or processors configured to access computer program code and operate as directed by the computer program code. The computer program is configured to cause the processor to perform a conversion between a visual media file and a bitstream of visual media data according to a format rule, and the format rule instructs the processor to determine one or more parameters of optical correction to be applied to at least one picture of the video and to determine one or more parameters based on a supplementary enhancement information (SEI) message associated with at least one picture.

[0011] A method and apparatus for video decoding in a decoder are provided, the method comprising: receiving a video bitstream including at least one picture of a video and supplementary enhancement information (SEI) messages associated with at least one picture; obtaining metadata from the SEI messages representing one or more parameters of an optical correction to be applied to at least one picture; and decoding the picture based on one or more parameters of the optical correction and applying the optical correction to the picture.

[0012] A video encoding method and apparatus are provided, the method comprising: receiving a video including at least one picture; determining an optical correction to be applied to at least one picture; and encoding at least one picture and a supplementary enhancement information (SEI) message associated with at least one picture, wherein the SEI message represents one or more parameters of the optical correction to be applied to at least one picture.

[0013] The SEI message may include the lens optical correction (loc) SEI message syntax of "lens_optical_correction".

[0014] Metadata may further indicate that one or more parameters of optical correction are applied based on any of the picture's radial distortion, chromatic aberration, and vignetting.

[0015] The loc SEI message syntax may further include a plurality of flags indicating whether the optical correction includes corrections for radial distortion, chromatic aberration, and vignetting, respectively.

[0016] The SEI message may further indicate the first focal center of the picture along the first axis, the second focal center of the picture along the second axis, and the focal length.

[0017] SEI messages can further indicate the lens model.

[0018] The SEI message may further represent a set of parameters of at least one model including one or more parameters, and optical correction is applied to at least one picture by at least one model. Brief explanation of the drawing

[0019] Additional features, nature, and various advantages of the disclosed subject matter will become more apparent through the following detailed description and attached drawings. FIG. 1 is a schematic example of a simplified block diagram of a communication system according to one or more embodiments. FIG. 2 is a schematic example of a simplified block diagram of a communication system according to one or more embodiments. FIG. 3 is a schematic example of a simplified block diagram of a decoder according to one or more embodiments. FIG. 4 is a schematic example of a simplified block diagram of an encoder according to one or more embodiments. FIG. 5 is a schematic example of a NAL unit and SEI header according to one or more embodiments. FIG. 6 is a schematic example of a source and an optically distorted image according to one or more embodiments. FIG. 7 is a schematic example of a model according to one or more embodiments. FIGS. 8a and FIGS. 8b are schematic examples of chromatic aberration according to one or more embodiments. FIG. 9 is a schematic example of a video transmission system according to one or more embodiments. FIG. 10 illustrates an example of parameters from a configuration file as a sample configuration of a lens optical correction SEI according to an embodiment. FIG. 11 illustrates an example (1100) in which radiation distortion correction according to an embodiment of the present specification is implemented. FIG. 12 illustrates an example in which correction of transverse chromatic aberration (TCA) is implemented according to an embodiment of the present specification. FIG. 13 is a schematic example of a computer system according to one or more embodiments. Specific details for implementing the invention

[0020] The proposed features discussed below may be used individually or combined in any order. Additionally, embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transient computer-readable medium.

[0021] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to one embodiment of the present disclosure. The communication system (100) may include at least two terminals (102, 103) interconnected via a network (105). For unidirectional transmission of data, the first terminal (103) may encode video data at a local location and transmit it to another terminal (102) via the network (105). The second terminal (102) may receive the coded video data from the other terminal from the network (105), decode the coded data, and display the restored video data. Unidirectional data transmission may be common in media service applications, etc.

[0022] FIG. 1 illustrates a second pair of terminals (101, 104) provided to support bidirectional transmission of coded video that may occur, for example, during a video conference. For bidirectional transmission of data, each terminal (101, 104) can code video data captured at a local location to transmit to another terminal via a network (105). Each terminal (101, 104) can also receive coded video data transmitted by another terminal, decode the coded data, and display the restored video data on a local display device.

[0023] In FIG. 1, terminals (101, 102, 103, and 104) may be illustrated as servers, personal computers, and smartphones, but the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure may be applied to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (105) represents any number of networks that transmit coded video data between terminals (101, 102, 103, and 104), for example, including wired and / or wireless communication networks. The communication network (105) may exchange data over circuit-switched and / or packet-switched channels. Typical networks include remote communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (105) may not be important to the operation of the present disclosure unless described below. The network (150) may include, for example, a Media Aware Network Element (MANE, 160) that may be included in a transmission path between terminals (130) and (140). The purpose of the MANE may be to selectively transmit a portion of media data to respond to network congestion, media switching, media mixing, archiving, and similar tasks typically performed by service providers rather than end users. These MANEs may parse and respond to a limited portion of the media transmitted through the network, such as, for example, syntactic elements related to the network abstraction layer of video coding technology or standards.

[0024] FIG. 2 illustrates the arrangement of a video encoder and a decoder in a streaming environment as an example of an application of the disclosed subject. The disclosed subject may be similarly applied to other video-supported applications, such as video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0025] The streaming system may include a capture subsystem (203) which may include a video source (201), for example, a digital camera, and may generate, for example, an uncompressed video sample stream (213). The amount of data in the sample stream (213) may be much greater than that in the encoded video bitstream and may be processed by an encoder (202) coupled to the camera (201). The encoder (202) may include hardware, software, or a combination thereof, thereby enabling or implementing the embodiments of the disclosed subject matter described in more detail below. An encoded video bitstream (204) with a smaller amount of data than that in the sample stream may be stored in a streaming server (205) for future use. One or more streaming clients (212, 207) may access the streaming server (205) to obtain copies (208, 206) of the encoded video bitstream (204). The client (212) may include a video decoder (211) that decodes a copy of an incoming encoded video bitstream (208) and generates an output video sample stream (210) that can be rendered to a display (209) or other rendering device (not shown). In some streaming systems, the video bitstreams (204, 206 and 208) may be encoded according to specific video coding / compression standards. Examples of such standards have been described above and are further described herein. Examples of such standards also include ITU-T recommendations H.265 and H.266. The disclosed subject matter may be used, for example, in the context of VVC.

[0026] FIG. 3 may be a functional block diagram of a video decoder (300) according to one embodiment of the present invention.

[0027] The receiver (302) may receive one or more codec video sequences to be decoded by the decoder (300); in the same or other embodiments, may receive one coded video sequence at a time, provided that the decoding of each coded video sequence is independent of other coded video sequences. The coded video sequences may be received from a channel (301) which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (302) may receive the encoded video data along with other data, such as, for example, encoded audio data and / or an additional data stream, which may be delivered to each user entity (not shown). The receiver (302) may separate the coded video sequences from the other data. To address network jitter, a buffer memory (303) may be coupled between the receiver (302) and the entropy decoder / parser (304) (hereinafter referred to as the "parser"). When the receiver (302) receives data from a storage / transmission device or an isochronous network having sufficient bandwidth and controllability, the buffer (303) may not be necessary or may be small. For use in a best-effort packet network such as the Internet, the buffer (303) may be necessary, may be relatively large, and advantageously may have an adaptive size.

[0028] The video decoder (300) may include a parser (304) for reconstructing symbols (313) from an entropy-coded video sequence. Categories of symbols include information used to manage the operation of the decoder (300) and potential information for controlling a rendering device, such as a display (312), which is not an integral part of the decoder but can be coupled thereto. The control information for the rendering device(s) may be in the form of supplementary enhancement information (SEI) messages or video usability information parameter set fragments (not shown). The parser (304) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow principles well known to a person skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (304) can extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to a group. The subgroups may include a picture group (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The entropy decoder / parser can also extract information such as transform coefficients, quantization parameter values, motion vectors, etc. from the coded video sequence.

[0029] The parser (304) can generate symbols (313) by performing an entropy decoding / parsing operation on a video sequence received from the buffer (303). The parser (304) can receive encoded data and selectively decode specific symbols (313). Additionally, the parser (304) can determine whether specific symbols (313) will be provided to a motion compensation prediction unit (306), a scaler / inverse transformation unit (305), an intra prediction unit (307), or a loop filter (311).

[0030] The reconstruction of the symbol (313) may include multiple different units depending on the type of the coded video picture or a part thereof (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors. Which units are included and in what manner may be controlled by subgroup control information parsed from the coded video sequence by the parser (304). For clarity, the flow of such subgroup control information between the parser (304) and the multiple units below is not illustrated.

[0031] In addition to the functional blocks already mentioned, the decoder (300) may be conceptually subdivided into a number of functional units as described below. In actual embodiments operating under commercial constraints, many of these units may interact closely with one another and may be integrated at least partially. However, for the purpose of illustrating the disclosed subject matter, the conceptual subdivision into the functional units below is appropriate.

[0032] The first unit is a scaler / inverse transform unit (305). The scaler / inverse transform unit (305) receives quantized transform coefficients as well as control information, including which transform to use as symbol(s) (313) from the parser (304), block size, quantization factor, quantization scaling matrix, etc. This can output a block containing sample values ​​that can be input to the aggregator (310).

[0033] In some cases, the output sample of the scaler / inverse transform (305) may be related to an intra-coded block; that is, a block that does not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. This prediction information may be provided by an intra-picture prediction unit (307). In some cases, the intra-picture prediction unit (307) uses surrounding already reconstructed information taken from the current (partially reconstructed) picture (309) to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator (310) adds the prediction information generated by the intra-prediction unit (307) to the output sample information provided by the scaler / inverse transform unit (305) on a sample basis.

[0034] In other cases, the output samples of the scaler / inverse unit (305) may be intercoded and potentially associated with a motion-compensated block. In this case, the motion compensation prediction unit (306) may access the reference picture memory (308) to obtain samples used for prediction. After motion-compensating the samples obtained according to the symbols (313) associated with the block, these samples may be added to the output of the scaler / inverse unit (in this case, referred to as residual samples or residual signals) by the aggregator (310) to generate output sample information. The address within the reference picture memory region from which the motion compensation unit obtains prediction samples may be controlled by a motion vector available to the motion compensation unit in the form of a symbol (313) that may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​obtained from the reference picture memory when a sub-sample precision motion vector is in use, a motion vector prediction mechanism, etc.

[0035] The output sample of the aggregator (310) may be subjected to various loop filtering techniques in the loop filter unit (311). The video compression technique may include an in-loop filter technique controlled by parameters included in the coded video bitstream and provided to the loop filter unit (311) as symbols (313) from the parser (304), but may also respond to meta-information obtained during decoding prior to a portion (decoding order) of the coded picture or coded video sequence, and to previously reconstructed and loop-filtered sample values.

[0036] The output of the loop filter unit (311) may be a sample stream that can be output to a display (312) which may be a render device, and may also be stored in a reference picture memory (557) for use in inter-picture prediction later.

[0037] Once fully reconstructed, a specific coded picture can be used as a reference picture for future prediction. Once the coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (304)), the current reference picture (309) can become part of the reference picture buffer (308), and a new current picture memory can be reallocated before initiating the reconstruction of the subsequent coded picture.

[0038] The video decoder (300) may perform decoding operations according to a predetermined video compression technique that may be documented in standards such as ITU-T Rec.H.265 and / or H.266. The coded video sequence may comply with the syntax defined by the video compression technique or standard used, in the sense that it complies with the syntax of the video compression technique or standard as defined in the video compression technique document or standard, particularly in the profile document therein. It may also be necessary to ensure compliance that the complexity of the coded video sequence is within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0039] In one embodiment, the receiver (302) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (300) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0040] FIG. 4 may be a functional block diagram of a video encoder (400) according to one embodiment of the present disclosure.

[0041] The encoder (400) can receive video samples from a video source (401) (not part of the encoder) capable of capturing video image(s) to be coded by the encoder (400).

[0042] A video source (401) may provide a source video sequence to be coded by an encoder (303) in the form of a digital video sample stream having any appropriate bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCb, RGB, ...), and any appropriate sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (401) may be a storage device that stores a pre-prepared video. In a video conferencing system, the video source (401) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed sequentially. The picture itself may consist of a spatial arrangement of pixels, wherein each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. A person skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0043] According to an embodiment, the encoder (400) can code and compress pictures of a source video sequence into a coded video sequence (410) in real time or under any other arbitrary time constraint as required by the application. Enforcing an appropriate coding rate is a function of the controller (402). The controller controls other functional units as described below and is functionally coupled to these units. The coupling is not illustrated for clarity. Parameters set by the controller may include rate control-related parameters (picture skip, quantizer, lambda value of rate-distortion optimization technique, ...), picture size, Group of Pictures (GOP) layout, maximum motion vector search range, etc. A person skilled in the art can easily identify other functions of the controller (402) that may be relevant to the video encoder (400) optimized for a particular system design.

[0044] Some video encoders operate in what a person skilled in the art would easily recognize as a “coding loop.” For an oversimplified description, the coding loop may consist of an encoding unit of the encoder (e.g., a source coder (403)) (which is responsible for generating symbols based on the input picture to be coded and reference picture(s)) and a (local) decoder (406) embedded in the encoder (400) that reconstructs the symbols to generate sample data in a manner similar to how a (remote) decoder performs (since all compression between the symbols and the coded video bitstream in the video compression technique considered in the disclosed subject is also lossless). The reconstructed sample stream is input into the reference picture memory (405). Since the decoding of the symbol stream results in bit-accurate results regardless of the location of the decoder (local or remote), the contents of the reference picture buffer are also bit-accurate between the local encoder and the remote encoder. In other words, the encoder's prediction unit "views" the exact same sample value as the reference picture sample as what the decoder sees when using the prediction during decoding. The basic principle of this reference picture synchronization (e.g., drift that occurs when synchronization cannot be maintained due to channel errors) is well known to those skilled in the art.

[0045] The operation of the “local” decoder (406) may be the same as the operation of the “remote” decoder (300) already described in detail above in relation to FIG. 3. However, with brief reference to FIG. 4, since the symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (408) and the parser (304) may be lossless, the entropy decoding part of the decoder (300), which includes the channel (301), receiver (302), buffer (303), and parser (304), may not be fully implemented in the local decoder (406).

[0046] One observation that can be made at this point is that any decoder technique, excluding parsing / entropy decoding present in the decoder, must also necessarily exist in the corresponding encoder in substantially the same functional form. Descriptions of encoder techniques may be omitted, as they are the inverse processes of the comprehensively described decoder techniques. More detailed explanations are required for specific areas, which are provided below.

[0047] As part of the operation, the source coder (403) may perform motion compensation predictive coding, which predictively codes an input frame by referencing one or more previously coded frames from a video sequence designated as a "reference frame." In this way, the coding engine (407) codes the difference between a pixel block of the input frame and a pixel block of the reference frame(s) that can be selected as a predictive reference for the input frame.

[0048] The local video decoder (406) can decode the coded video data of a frame that can be designated as a reference frame based on the symbols generated by the source coder (403). The operation of the coding engine (407) may advantageously be a lossy process. If the coded video data can be decoded by a video decoder (not shown in FIG. 4), the reconstructed video sequence may typically be a copy of the source video sequence that has some errors. The local video decoder (406) can replicate the decoding process that can be performed by the video decoder on the reference frame and allow the reconstructed reference frame to be stored in the reference picture memory (405) (e.g., a cache). In this way, the encoder (400) can locally store a copy of the reconstructed reference frame having common content, such as the reconstructed reference frame (where there are no transmission errors) that will be acquired by the remote video decoder.

[0049] The predictor (404) can perform a predictive search for the coding engine (407). That is, for a new frame to be coded, the predictor (404) can search the reference picture memory (405) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., which can be used as appropriate predictive references for the new picture. The predictor (404) can operate on a sample block-by-pixel block basis to find appropriate predictive references. In some cases, as determined by the search results obtained by the predictor (404), the input picture may have predictive references derived from multiple reference pictures stored in the reference picture memory (405).

[0050] The controller (402) can manage the coding operation of the video coder (403), for example, including the setting of parameters and subgroup parameters used to encode video data.

[0051] The output of all the aforementioned functional units can be entropy-coded in an entropy coder (408). The entropy coder converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding.

[0052] The transmitter (409) may buffer the coded video sequence(s) generated by the entropy coder (408) to prepare for transmission through the communication channel (411), and the communication channel (411) may be a hardware / software link to a storage device to store the encoded video data. The transmitter (409) may merge the coded video data from the video coder (403) with other data to be transmitted, such as, for example, coded audio data and / or an additional data stream (source not shown).

[0053] The controller (402) can manage the operation of the encoder (400). During coding, the controller (405) can assign a specific coded picture type to each coded picture, which can affect the coding technique that can be applied to each picture. For example, a picture may frequently be assigned to one of the following frame types.

[0054] An intra-picture (I-picture) may be one that can be coded and decoded without using any other frames within the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, independent decoder refresh pictures. A person skilled in the art is aware of such variations of I-pictures and their respective applications and characteristics.

[0055] The prediction picture (P picture) may be coded and decoded using inter-prediction or intra-prediction, which utilizes at most one motion vector and reference index to predict the sample value of each block.

[0056] A bidirectional prediction picture (Picture B) may be coded and decoded using inter-prediction or intra-prediction, which uses up to two motion vectors and reference indices to predict sample values ​​of each block. Similarly, a multi-prediction picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0057] Typically, a source picture can be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively) and coded block by block. Blocks can be predictively coded by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, a block of picture I can be non-predictively coded or predictively coded by referencing an already coded block of the same picture (spatial prediction or intra prediction). A pixel block of picture P can be coded non-predictively, through spatial prediction, or through temporal prediction referencing one previously coded reference picture. A block of picture B can be coded non-predictively, through spatial prediction, or through temporal prediction referencing one or two previously coded reference pictures.

[0058] The video coder (400) can perform coding operations according to a predetermined video coding technology or standard, such as ITU-T Rec.H.265 and / or H.266. During the operation, the video coder (400) can perform various compression operations, including predictive coding operations that utilize the temporal and spatial redundancy of the input video sequence. Thus, the coded video data can comply with the syntax defined by the video coding technology or standard used.

[0059] In one embodiment, the transmitter (409) may transmit additional data along with the encoded video. The source coder (403) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, supplementary enhancement information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0060] Compressed video may be augmented within the video bitstream by supplementary enhancement information, for example, in the form of supplementary enhancement information (SEI) messages or video usability information (VUI). A video coding standard may include specification sections for SEI and VUI. SEI and VUI information may also be specified in independent specifications that may be referenced by the video coding specification.

[0061] According to the embodiments of the present specification, the encoding and decoding processes can each be regarded as processing of visual media data that performs a conversion between a visual media file and a bitstream of visual media data according to format rules.

[0062] Referring to FIG. 5, an example (500) of an exemplary layout of a coded video sequence (CVS) according to H.266 and an embodiment of the present specification is shown. The coded video sequence is subdivided into Network Abstraction Layer units (NAL units). An exemplary NAL unit (501) may include a NAL unit header (502), which is again composed of 16 bits as follows: forbidden_zero_bit (503) and nuh_reserved_zero_bit (504) may not be used in H.266 and may be 0 in an H.266 compliant NAL unit. The 3 bits of nuh_layer_id (505) may indicate the layer (spatial, SNR, or multiview enhancement) to which the NAL unit belongs. The 5 bits of nuh_nal_unit_type (506) define the type of the NAL unit. In H.266 according to the embodiment of this specification, 22 NAL unit type values ​​are defined for NAL unit types, 6 NAL unit types are reserved, and 4 NAL unit type values ​​are unspecified and may be used by standards other than H.266. Finally, 3 bits of the NAL unit header (502) indicate the temporal layer nuh_temporal_id_plus1 (506) to which the NAL unit belongs.

[0063] A coded picture may contain one or more Video Coding Layer (VCL) NAL units and zero or more non-VCL NAL units. VCL NAL units may contain coded data that conceptually belongs to the Video Coding Layer as previously introduced. Non-VCL NAL units may contain data that conceptually belongs to data that does not conceptually belong to the Video Coding Layer. Taking H.266 as an example, these units may be classified into parameter sets, picture headers, NAL units, prefix and suffix SEI NAL unit types, filler data NAL unit types (FD_NUT), and reserved and unspecified NAL unit types.

[0064] (1) A set of parameters that may contain information that may be necessary for the decoding process and can be applied to more than one coded picture. The parameter set and conceptually similar NAL units may be types of NAL units such as DCI_NUT (Decoding Capability Information, DCI), VPS_NUT (Video Parameter Set, VPS, which sets layer relationships among various items), SPS_NUT (Sequence Parameter Set, SPS, which sets parameters that are used and maintained throughout the coded video sequence (CVS) among various items), PPS_NUT (Picture Parameter Set, PPS, which sets parameters that are used and maintained within the coded picture among various items), and PREFIX_APS_NUT and SUFFIX_APS_NUT (Prefix and Suffix Adaptation Parameter Sets). Since the parameter set may contain information necessary for the decoder to decode the VCL NAL unit, it is referred to herein as a "normative" NAL unit.

[0065] (2) Also, a picture header (PH_NUT) which is a “normative” NAL unit.

[0066] (3) NAL units that mark specific locations within a NAL unit stream. These include NAL units with NAL unit types such as AUD_NUT (access unit separator), EOS_NUT (end of sequence), and EOB_NUT (end of bitstream). These are non-normative and also informative, as it is known that a compliant decoder does not require them for the decoding process, but must be able to receive them in the NAL unit stream.

[0067] (4) Prefix and suffix SEI NAL unit types (PREFIX_SEI_NUT and SUFFIX_SEI_NUT) representing NAL units containing prefix and suffix supplemental enhancement information. In H.266 (04 / 2022), NAL units are informative because they are not required in the decoding process.

[0068] (5) Filler data NAL unit type (FD_NUT) represents filler data; data that may be random and may be used to “waste” bits in a NAL unit stream or bitstream, which may be required for transmission through a specific isochronous transmission environment.

[0069] (6) Reserved and non-designated NAL unit types.

[0070] Referring again to FIG. 5, a layout of a NAL unit stream in a decoding sequence (510) containing a coded picture (511) containing some types of NAL units introduced earlier is shown. At some point in the beginning of the NAL unit stream, DCI (512), VPS (513), and SPS (514) may be combined to set parameters that a decoder can use to decode a coded picture of a coded video sequence (CVS) containing a coded picture (511) of the NAL unit stream.

[0071] The coded picture (511) may include a prefix APS 516, a picture header (PH, 517), a prefix SEI (518), one or more VCL NAL units (519), and a suffix SEI (520) in the illustrated order or any other order corresponding to the video coding technology or standard in use (here, H.266).

[0072] Prefix and suffix SEI NAL units (518, 520) were introduced during the standard development process because, for some SEI messages, the content of the message is known before the coding of a given picture begins, whereas for others, the content is known only after the picture is coded. Allowing specific SEI messages to appear early or late in the NAL unit stream of a picture coded via prefix and suffix SEIs enables buffering to be avoided. As one example, since the sampling time of the picture to be coded by the encoder is known before the picture is coded, the picture timing SEI message may be a prefix SEI message (516). Meanwhile, the decoded picture hash SEI message, which contains the hash of the sample values ​​of the decoded picture and can be used, for example, to debug the encoder implementation, is a suffix SEI message (518) because the encoder cannot calculate the hash for the reconstructed sample before the picture is coded. The location of prefix and suffix SEI NAL units may not be limited to that location within the NAL unit stream. The terms "prefix" and "suffix" may refer to which coded picture or NAL unit a prefix / suffix SEI message may be associated, and details of such applicability may be specified, for example, in the semantic description of a given SEI message.

[0073] Referring again to FIG. 5, a simplified syntax diagram of a NAL unit containing a prefix or suffix SEI message (520) is illustrated. This syntax may be a container format for multiple SEI messages that can be returned within a single NAL unit. For clarity, details of the extension mechanism for both the payload size and payload type number ranges specified in H.266 are omitted here. Like other NAL units, the SEI NAL unit begins with a NAL unit header (521). Following the header are one or more SEI messages; two of which (530, 531) are illustrated and described below. Each SEI message within the SEI NAL unit includes: an 8-bit payload_type_byte (522) specifying one of 256 different SEI types (or extension indicators); and an 8-bit payload_size_byte (523) specifying the number of bytes of the SEI payload (or the presence of an extension block); and may include the number of bytes of payload_size-bytes minus 1 of the number of bytes of payload (524). The syntax of the payload (524) depends on the SEI message and may have any length between 0 bytes and 254 bytes unless an extension mechanism is used (not shown), and if an extension mechanism is used, the syntax does not limit the payload size.

[0074] Referring to FIG. 9, a video distribution chain according to the present embodiment is illustrated. A camera (901) may include a lens (902) or other optical element that generates distortion. The lens may be directed toward an object (903). An image captured by a sensor (905) may be distorted by effects such as barrel distortion, pincushion distortion, chromatic aberration, and vignetting distortion, but is not limited thereto. As a possible example, it may be a distorted image (906) captured by the sensor. The distorted image (906) may be fed to an encoder (907) to generate a bitstream (908) as a result. The bitstream (908) may include specific metadata coded, for example, in the form of an SEI message (909). As the camera (901) is coupled to the encoder (907), the encoder may recognize the characteristics of the camera including distortion that the camera lens (902) may generate. The encoder may include camera characteristics including lens characteristics in the form of one or more SEI messages (909).

[0075] For example, note that lens distortion may vary over time, such as when a zoom lens is employed and the user adjusts the zoom magnification. For instance, while substantial distortion may not exist when the zoom lens is set to magnify distant details, significant distortion may exist when the zoom lens is set to capture an ultra-wide-angle scene in a fisheye setting.

[0076] According to embodiments of this specification, many implementation-dependent mechanisms may be used to combine a camera and an encoder to transmit camera characteristics, including lens characteristics, such as, for example, a serial link and a standardized or proprietary protocol for transmitting camera characteristics, or including camera characteristics in metadata associated with each picture to be encoded. In this regard, a mechanism capable of transmitting camera characteristics (updated or original) associated with any given coded picture within a video bitstream is suitable, and a mechanism for implementing metadata with picture-specific fineness may be an SEI message.

[0077] Some cameras may include hardware or software that converts potentially distorted images into an isolinear representation suitable for uncorrected displays. However, since performing distortion correction in a camera may have certain disadvantages, this is not universally implemented. A first disadvantage may be the additional cost and power load of the processing hardware / software, which may be difficult to accept for certain cameras, such as cost-effective, battery-powered, and lightweight action cameras. A second disadvantage may be that the compensated picture may not be rectangular, or may include artificially generated samples unrelated to the captured object, even if padded to be rectangular. Other disadvantages are also apparent to those skilled in the art upon reviewing the present disclosure.

[0078] Referring again to FIG. 9, a bitstream (908) containing SEI message(s) (909) may be stored, for example, on a server (910) or locally in a memory device, or delivered directly to a decoder (911). The decoder may receive the bitstream (908) accordingly and decode it into a series of reconstructed pictures (912) (only one reconstructed picture is shown) and associated metadata (913). The associated metadata may include camera characteristic metadata. The decoder may deliver the camera metadata to a renderer (914). The exact mechanism of such delivery may likewise vary depending on the implementation. Delivery may be made via a network, a serial link, or by including it in the metadata associated with each picture. Additionally, in some cases, the decoder (911) and the renderer (914) may be closely integrated within the same hardware / software structure, so that the data structure reconstructed by the decoder (911), including metadata, can be directly accessed by the renderer (914).

[0079] The renderer (914) can produce a corrected and visually satisfactory output picture (915) that is displayed on a display (916) or used in other ways by modifying the reconstructed picture (912) using information passed to the metadata (913).

[0080] Below, three exemplary types of distortion and mechanisms to address them are introduced. These mechanisms can be implemented and applied individually, sequentially in any order, or in combination.

[0081] For example, embodiments of this specification provide ultra-wide angle and fisheye camera characteristics. Fisheye lenses can sometimes have a very wide field of view exceeding 180 degrees. They can capture a spherical or hemispherical view that includes a significant amount of distortion. This distortion may result in either barrel distortion (outward curvature) or pincushion distortion (inward curvature) near the edges of the frame. Fisheye lenses can also have very short focal lengths. The shorter the focal length, the greater the potential depth of field.

[0082] Ultra-wide-angle lenses also have a wide field of view, but they do not reach the extreme degree of coverage typically seen in fisheye lenses. They typically provide a field of view between 90 and 120 degrees.

[0083] FIG. 6 illustrates an example (600) of distortion that may occur when using an ultra-wide angle or fisheye lens. Consider the image on the right (601) (of a sequence of images) captured by a fisheye lens, with a human head behind a square mesh. The image is visually distorted; it is not square, the head part is bulged in the center, and the edges are excessively narrow, etc. After correction using appropriate parameters, the corrected image on the left (602) is natural and visually pleasing.

[0084] Radial distortion introduced by a wide-angle optical system can be corrected using various correction models. Referring to FIG. 7, the model [denoted r] of the distorted image (denoted r d ) and undistorted images (denoted r u It represents the relationship (701) between the distance from the focal point (typically the center of the image) to a point (pixel).

[0085] In this regard, notable models include the Poly3 model, Poly5 model, Ptlens model, and Adobe Camera Model (ACM) according to the embodiments of this specification.

[0086] The Poly3 model is a third-order polynomial model and is defined as follows.

[0087]

[0088] Here, k1 is the correction term. The value of k1 is within the range [-1,1], with a default value of 0 (i.e., no distortion). When k1 > 0, barrel distortion is observed, whereas k1 < 0 corresponds to the pincushion effect.

[0089] The Poly5 model is a fifth-order polynomial model defined as follows.

[0090]

[0091] Here, k1 is a second-order correction term, and k2 is a double second-order term. The values ​​of k1 and k2 are in the range [-1,1], and the default value is 0 (i.e., no distortion).

[0092] The Ptlens model is a more sophisticated model expressed as follows.

[0093]

[0094] Here, "a", "b", and "c" are parameters corresponding to the physical characteristics of the lens within the range [-1,1], and are essentially equal to 0 (i.e., no correction).

[0095] The Adobe Camera Model (ACM) is a more complex representation model of lens distortion that has different coordinate systems in which parameters are applied in units of lens focal length. The model is defined as follows:

[0096]

[0097] In this case, the corresponding attributes are called k1, k2, k3, k4, and k5, and are set to 0 by default.

[0098] The ACM model is suitable for correcting fisheye lenses using only the k1 and k2 parameters.

[0099] By transmitting an appropriate set of the aforementioned parameters, the image can be corrected in the renderer. The syntax and semantics of SEI messages covering the model, along with various related aberrations that can be displayed, are described below.

[0100] For example, consider chromatic aberration.

[0101] Light is electromagnetic radiation, and its characteristics are defined primarily by wavelength. As illustrated in the example (800A) of FIG. 8a, when rays of different colors pass through the edges of a lens (801), the rays may be refracted at different angles depending on the wavelength due to the prism effect. For example, the red light component (803) of white light (802) may be refracted at a steeper angle than the green component (804), and the green component may be refracted at a steeper angle than the blue component (805). The characteristics of this prism effect depend on the type of material the lens is made of, among other possible factors.

[0102] The example (800B) of FIG. 8b illustrates an example of chromatic aberration. The source (810) may be black and white line art with uniform line thickness. The picture (811) may be the result of capturing with a lens that has chromatic aberration distortion primarily in the horizontal direction. Here, the horizontal line (812) may have a "shade" that appears blue when rendered in color. In the black and white rendering of this specification, the shadow is difficult to see. However, the enlarged view (813) illustrates a black line (814) and a blue shade (815).

[0103] The effect is the cause of chromatic aberration, also known as color bleeding, when colors begin to separate at the edges of the picture. Transverse chromatic aberration (TCA) correction can compensate for the prism effect. Additionally, according to embodiments of this specification, there are several models that take the green component as a reference and align the red and blue components to it (e.g., linear TCA model, cubic polynomial TCA model, and Adobe Camera Model TCA model).

[0104] The linear TCA model can be expressed as follows.

[0105]

[0106] Here, r d,R and r d,B are the red and blue coordinates of a point within the distorted image, respectively, and r u,R and r u,B are the red and blue coordinates in the distortion-corrected undistorted image, respectively, and k r and k b is a TCA parameter with a default value of 1.

[0107] The third-order polynomial TCA model can be expressed as follows.

[0108]

[0109] Here, r d,R and r d,B are the red and blue coordinates of a point within the distorted image, respectively, and r u,R and r u,B are the red and blue coordinates in the distortion-corrected undistorted image, respectively, and b R , c R , v R , b B , v B , and v B is a TCA parameter with a default value of 1.

[0110] The Adobe camera model TCA model can be expressed as follows.

[0111]

[0112] Here, , (x d,R ,y d,R ) and (x d,B ,y d,B ) are the red and blue coordinates of a point in the distorted image, respectively, and r u,R and r u,B are the red and blue coordinates in the distortion-corrected undistorted image, respectively, and b R , c R , v R , b B , v B and v B is a TCA parameter with a default value of 1.

[0113] The embodiments described herein also take vignetting into account. Vignetting is the effect of reduced light intensity at the corners of an image due to lens characteristics. The vignetting effect can be modeled as a dome applied to the picture. In some cases, the amount of vignetting is not noticeable. However, even a small amount of vignetting can become a problem after the video is stitched in 360 mode, such as when dark bands appear distinctly at the stitching edges, in mechanical processing scenarios, or in certain other applications. Vignetting correction can be performed only on the brightness distribution within the video, for example, using one of the widely used models (e.g., the Pablo D'Angelo (PA) vignetting model and / or Adobe's (ACM) vignetting model).

[0114] Pablo D'Angelo's (PA) vignette model can be described as follows.

[0115]

[0116] Here, C d represents the corrected target image pixel brightness, and C s represents the brightness of the uncorrected source image pixels, and k1, k2, and k3 represent the polynomial parameters of the correction.

[0117] Adobe's Vignetting Model (ACM) can be specified as follows.

[0118]

[0119] Here, C d represents the corrected target image pixel brightness, and C s represents the brightness of the uncorrected source image pixels, and α1, α2, and α3 represent the parameters of correction expressed in units of focal length.

[0120] Information regarding the above model may be encoded as metadata in the encoded video bitstream. Since lens parameters may change frequently—for example, when a zoom lens is used—it may be desirable to use a metadata structure that allows for the association of lens parameters at the encoded picture level. Accordingly, one suitable solution may be an SEI message according to the embodiments of this specification.

[0121] The following describes the syntax and semantics of an SEI message that enables the transmission of information about the model reviewed above, in accordance with the embodiments of this specification. A person skilled in the art can easily derive variations of this SEI message that support only a subset of such models, divide a single SEI message into multiple SEI messages that encompass different models or sets of models, and include the presented information in other metadata structures, such as visual usability information structures, parameter sets, or any other syntactic structures.

[0122] The syntax is presented in the form of a table in a format familiar to those familiar with ITU and MPEG video codecs. Specifically, C-style syntax is used on the right side of the table to indicate when a specific syntax element is present in the bitstream. The names of the syntax elements are indicated in bold. The descriptor column specifies the data type for the syntax element. For example, u(1) means a 1-bit unsigned integer or flag, u(16) means a 16-bit unsigned integer, and the same applies hereinafter. Since the syntax table primarily uses unsigned integers of appropriate length, it has an appropriate number range accordingly. A person skilled in the art can easily determine where other data types, such as variable-length integers, may be more appropriate from the perspective of coding efficiency, and can substitute such data types for fixed-length integers. Furthermore, Table 1 below shows the lens optical correction SEI message syntax according to one or more embodiments.

[0123] 표 1 - 렌즈 광학 보정 SEI 메시지 구문

[0124]

[0125]

[0126] SEI messages based on syntax may use the meanings described below. In other words, the following definitions of meaning follow conventions used in ITU / MPEG standards, which include the use of prescriptive or informational language defined by ITU / ISO / IEC for use in such standards.

[0127] This SEI message provides a lens distortion model to the decoder, enabling image correction.

[0128] loc_cancel_flag A value equal to 1 indicates that the SEI message cancels the persistence of any previous lens optical correction SEI message applied to the current layer in output order. A value equal to 0 indicates that lens optical correction follows. loc_focal_parameters_present_flag A value equal to 1 indicates that the lens's focus parameters exist within the lens optical correction SEI message syntax. A value equal to 0 indicates that no focus parameters exist. loc_focal_center_x defines the horizontal position of the focus in the picture. This is basically half the width of the picture. loc_focal_center_y It defines the vertical position of the focus in the picture. This is basically half the height of the picture. loc_focal_length represents the focal length of the lens, that is, the distance from the lens where light rays incident on the lens parallel to the optical axis converge. This is expressed in millimeters, and the default value is set to 0, meaning unknown.

[0129] loc_radial_distortion_parameters_present_flag A value equal to 1 indicates that radial distortion parameters are present within the lens optical correction SEI message syntax. A value equal to 0 indicates that radial correction is not expected. loc_radial_distortion_model_id It identifies the types of radiation distortion models specified in Table 1.

[0130] The value of loc_radial_distortion_model_id must be within the range of 0 to 2. Values ​​greater than 3 are reserved for future use by ITU T | ISO / IEC and must not exist in bitstreams compliant with this version of the specification. The decoder must ignore lens optical correction SEI messages where loc_radial_distortion_model_id is greater than 3.

[0131] 표 2 - loc_radial_distortion_model_id

[0132]

[0133] When loc_radial_distortion_model_id is equal to 0, the 3rd-order polynomial model is specified as follows, and

[0134]

[0135] Here, r d represents the distorted radius (normalized distance to the image center), and r u represents the corrected non-distortion radius, and k1 is the distortion coefficient.

[0136] When loc_radial_distortion_model_id is equal to 1, the 5th-order polynomial model is specified as follows, and

[0137]

[0138] Here, r d represents the distorted radius (normalized distance to the image center), and r u represents the corrected non-distortion radius, and k1 and k2 are distortion coefficients.

[0139] When loc_radial_distortion_model_id is equal to 2, the Ptlens model is specified as follows, and

[0140]

[0141] Here, r d represents the distorted radius (normalized distance to the image center), and r u represents the corrected non-distortion radius, and a, b, and c are distortion coefficients.

[0142] When loc_radial_distortion_model is equal to 3, the ACM model is specified as follows, and

[0143]

[0144] Here, am.

[0145] Here, (x d ,y d ) is the coordinates of a point within the distorted image, and (x u ,y u ) is the coordinate of the same point in the undistorted image, and k1, k2, k3, k4, and k5 represent the parameters of radial distortion, which are equal to 0 by default.

[0146] rd_poly3_k1_times1000 represents 1000 times the k1 parameter value of the Poly3 radiation distortion model. rd_poly3_k1_times1000 has a value between -1000 and 1000, and the default value is set to 0. rd_poly5_k1_times1000 represents 1000 times the k1 parameter value of the Poly5 radial distortion model. rd_poly5_k1_times1000 takes a value between -1000 and 1000, and the default value is set to 0. r d_poly5_k2_times1000 represents 1000 times the k1 parameter value of the Poly5 radial distortion model. rd_poly5_k2_times1000 has a value between -1000 and 1000, and the default value is set to 0. rd_ptl_a_times1000 represents 1000 times the value of the 'a' parameter of the Ptlens radial distortion model. rd_ptl_a_times1000 has a value between -1000 and 1000, and the default value is set to 0. rd_ptl_b_times1000 represents 1000 times the value of the 'b' parameter of the Ptlens radial distortion model. rd_ptl_b_times1000 has a value between -1000 and 1000, and the default value is set to 0.

[0147] rd_ptl_c_times1000 represents 1000 times the value of the 'c' parameter of the Ptlens radial distortion model. rd_ptl_c_times1000 has a value between -1000 and 1000, and the default value is set to 0. rd_acm_k1_times1000 represents 1000 times the k1 parameter value of the ACM radiation distortion model. rd_acm_k1_times1000 has a value between -1000 and 1000, and the default value is set to 0. rd_acm_k2_times1000 represents 1000 times the k2 parameter value of the ACM radiation distortion model. rd_acm_k2_times1000 has a value between -1000 and 1000, and the default value is set to 0.

[0148] rd_acm_k3_times1000 represents 1000 times the k3 parameter value of the ACM radiation distortion model. rd_acm_k3_times1000 has a value between -1000 and 1000, and the default value is set to 0. rd_acm_k4_times1000 represents 1000 times the k4 parameter value of the ACM radiation distortion model. rd_acm_k4_times1000 has a value between -1000 and 1000, and the default value is set to 0. rd_acm_k5_times1000 represents 1000 times the k5 parameter value of the ACM radiation distortion model. rd_acm_k5_times1000 has a value between -1000 and 1000, and the default value is set to 0.

[0149] loc_tca_parameters_present_flag A value equal to 1 indicates that the transverse chromatic aberration parameter is present in the lens optical correction SEI message syntax. A value equal to 0 indicates that transverse chromatic aberration correction is not expected. loc_tca_model_id Identifies the type of radiation distortion model as specified in Table X. The value of loc_tca_model_id must be in the range of 0 to 2. Values ​​greater than 2 are reserved for future use by ITUT | ISO / IEC and must not exist in bitstreams compliant with this version of the specification. The decoder must ignore lens optical correction SEI messages where loc_tca_model_id is greater than 2.

[0150] 표 3 - loc_tca_model_id

[0151]

[0152] When loc_tca_model_id is equal to 0, the linear model is specified as follows, and

[0153]

[0154] The attribute is "k r " and "k b It is referred to as ". Their default value is 1.

[0155] When loc_tca_model_id is equal to 1, the linear model is specified as follows, and

[0156]

[0157] The attributes are "b r ", "c r ", "v r ", "b b ", "c b " and "v b It is referred to as ". v r and v b The default value of is 1.

[0158] When loc_tca_model_id is equal to 2, the ACM model is specified as follows, and

[0159]

[0160] Here,

[0161] The corresponding XML attributes are referred to as "alpha0" through "alpha5" and "beta0" through "beta5". The default values ​​for alpha0 and beta0 are 1, and the rest are 0.

[0162] tca_kr_times1000 is k of the linear transverse chromatic aberration model r Represents 1000 times the parameter value. kr_tca_times1000 is set to 1 by default. tca_kb_times1000 is k of the linear transverse chromatic aberration model b Represents 1000 times the parameter value. kb_tca_times1000 is set to 1 by default. tca_br_times1000 b of the Poly3 transverse chromatic aberration model r Represents 1000 times the parameter value. tca_br_times1000 is set to 1 by default. tca_cr_times1000 c of the Poly3 transverse chromatic aberration model r Represents 1000 times the parameter value. tca_cr_times1000 is set to 1 by default. tca_vr_times1000 v of the Poly3 transverse chromatic aberration model r Represents 1000 times the parameter value. tca_vr_times1000 is set to 1 by default. tca_bb_times1000 b of the Poly3 transverse chromatic aberration model b Represents 1000 times the parameter value. tca_br_times1000 is set to 1 by default.

[0163] tca_cb_times1000 c of the Poly3 transverse chromatic aberration model b Represents 1000 times the parameter value. tca_cr_times1000 is set to 1 by default. tca_vb_times1000 v of the Poly3 transverse chromatic aberration model b Represents 1000 times the parameter value. tca_vr_times1000 is set to 1 by default. tca_alpha0_times1000 represents 1000 times the α0 parameter value of the ACM transverse chromatic aberration model. tca_alpha0_times1000 is set to 1 by default. tca_alpha1_times1000 represents 1000 times the α1 parameter value of the ACM transverse chromatic aberration model. tca_alpha1_times1000 is set to 0 by default.

[0164] tca_alpha2_times1000 represents 1000 times the α2 parameter value of the ACM transverse chromatic aberration model. tca_alpha2_times1000 is set to 0 by default. tca_alpha3_times1000 represents 1000 times the α3 parameter value of the ACM transverse chromatic aberration model. tca_alpha3_times1000 is set to 0 by default. tca_alpha4_times1000 represents 1000 times the α4 parameter value of the ACM transverse chromatic aberration model. tca_alpha4_times1000 is set to 0 by default. tca_alpha5_times1000 represents 1000 times the α5 parameter value of the ACM transverse chromatic aberration model. tca_alpha5_times1000 is set to 0 by default. tca_beta0_times1000 represents 1000 times the β0 parameter value of the ACM transverse chromatic aberration model. tca_beta0_times1000 is set to 1 by default.

[0165] tca_beta1_times1000 represents 1000 times the β1 parameter value of the ACM transverse chromatic aberration model. tca_beta1_times1000 is set to 0 by default. tca_beta2_times1000 represents 1000 times the β2 parameter value of the ACM transverse chromatic aberration model. tca_beta2_times1000 is set to 0 by default. tca_beta3_times1000 represents 1000 times the β3 parameter value of the ACM transverse chromatic aberration model. tca_beta3_times1000 is set to 0 by default. tca_beta4_times1000 represents 1000 times the β4 parameter value of the ACM transverse chromatic aberration model. tca_beta4_times1000 is set to 0 by default. tca_beta5_times1000 represents 1000 times the β5 parameter value of the ACM transverse chromatic aberration model. tca_beta5_times1000 is set to 0 by default.

[0166] loc_vignetting_parameters_present_flag A value equal to 1 indicates that vignetting parameters are present in the lens optical correction SEI message syntax. A value equal to 0 indicates that vignetting correction is not expected. loc_vignetting_model_id Identifies the type of vignetting model specified in Table 3. The value of loc_vignetting_model_id must be within the range of 0 to 1. Values ​​greater than 1 are reserved for future use by ITUT | ISO / IEC and must not exist in bitstreams compliant with this version of the specification. The decoder must ignore lens optical correction SEI messages where loc_vignetting_model_id is greater than 1.

[0167] 표 4 - loc_vignetting_model_id

[0168]

[0169] When loc_vignetting_model_id is equal to 0, the Pablo D'Angelo vignetting model is specified as follows, and

[0170]

[0171] Here, C d represents the corrected target image pixel brightness, and C s represents the uncorrected source image pixel brightness, and k1, k2, and k3 represent the polynomial parameters of the correction. k1, k2, and k 3의 The default value is 0.

[0172] When loc_vignetting_model_id is equal to 1, the ACM vignette model is specified as follows, and

[0173]

[0174] Here, C d represents the corrected target image pixel brightness, and C srepresents the brightness of the uncorrected source image pixels, and α1, α2, and α3 represent the correction parameters expressed in units of focal length. The default values ​​for α1, α2, and α3 are 0.

[0175] vi_pa_k1_times1000 represents 1000 times the k1 parameter value of the Pablo D'Angelo vignette model. vi_pa_k1_times1000 is set to 0 by default. vi_pa_k2_times1000 represents 1000 times the k2 parameter value of the Pablo D'Angelo vignette model. vi_pa_k2_times1000 is set to 0 by default. vi_pa_k3_times1000 represents 1000 times the k3 parameter value of the Pablo D'Angelo vignette model. vi_pa_k3_times1000 is set to 0 by default. vi_acm_alpha1_times1000 represents 1000 times the α1 parameter value of the ACM vignetting model. vi_acm_alpha1_times1000 is set to 0 by default. vi_acm_alpha2_times1000 represents 1000 times the α2 parameter value of the ACM vignetting model. vi_acm_alpha2_times1000 is set to 0 by default. vi_acm_alpha3_times1000 represents 1000 times the α3 parameter value of the ACM vignetting model. vi_acm_alpha3_times1000 is set to 0 by default.

[0176] loc_persistence_flag loc_persistence_flag specifies the persistence of the lens optical correction SEI message for the current layer. If loc_persistence_flag is equal to 0, it specifies that the lens optical correction SEI message applies only to the currently decoded picture. If loc_persistence_flag is equal to 1, it specifies that the lens optical correction SEI message applies to the currently decoded picture and persists to all subsequent pictures of the current layer in output order until one or more of (i) a new CLVS of the current layer starts, (ii) the bitstream ends, or (iii) the picture of the current layer in the AU associated with the lens optical correction SEI message is output after the current picture in output order is true.

[0177] In addition, in addition to Table 1, Table 4 below shows other lens optical correction SEI message syntax according to one or more embodiments.

[0178] 표 4 - 렌즈 광학 보정 SEI 메시지 구문

[0179]

[0180]

[0181] The description below provides the meaning of the lens optical correction SEI message regarding the syntax of the embodiments of the present specification, such as that in Table 4 above.

[0182] The lens optical correction SEI message enables image correction by providing a lens distortion model to the decoder. If parameters for multiple distortion types are available in the lens optical correction SEI message, the parameters are expressed with the intention of correction in the following sequential order: first, devenetting (if present), then anti-TCA (if present), and then inverse transformation of radiative distortion (if present).

[0183] According to an embodiment, the parameter is expressed so that the correction can be applied to a sample in a linear RGB format. A picture expressed as YUV sample data (e.g., Y'CbCr) can be converted to linear RGB to apply lens correction as intended.

[0184] The parameters of this SEI message are applied to the cropped and decoded picture.

[0185] If a frame packing array SEI message exists in CLVS, the parameters of this SEI message are applied to each constituent frame of the frame packing array SEI message.

[0186] loc_cancel_flag A value equal to 1 indicates that the SEI message cancels the persistence of any previous lens optical correction SEI message applied to the current layer in output order. A value equal to 0 indicates that lens optical correction follows. loc_persistence_flag loc_persistence_flag specifies the persistence of the lens optical correction SEI message for the current layer. If loc_persistence_flag is equal to 0, it specifies that the lens optical correction SEI message applies only to the currently decoded picture. If loc_persistence_flag is equal to 1, it specifies that the lens optical correction SEI message applies to the currently decoded picture and persists to all subsequent pictures of the current layer in output order until one or more of the following are true: (i) a new CLVS of the current layer starts, (ii) the bitstream ends, or (iii) the picture of the current layer in the AU associated with the lens optical correction SEI message is output after the current picture in output order.

[0187] loc_lens_model_present_flag A value equal to 1 indicates that the loc_lens_model syntax element exists within the lens optical correction SEI message syntax. A value equal to 0 indicates that the loc_lens_model syntax element does not exist within the lens optical correction SEI message syntax. loc_reserved_zero_5bits must be equal to 0 in a bitstream suitable for this version of the specification. Greater than 0 values ​​for loc_reserved_zero_5bits are reserved for future use by ITU-T | ISO / IEC. The decoder must allow the appearance of a greater than 0 value for loc_reserved_zero_5bits in the bitstream and must ignore the value of loc_reserved_zero_5bits. loc_lens_model Specifies a string describing the applicable lens model. loc_focal_parameters_present_flag A value equal to 1 indicates that the lens's focus parameters exist within the lens optical correction SEI message syntax. A value equal to 0 indicates that no focus parameters exist.

[0188] Defines the horizontal position of the focus within the picture, expressed as a ratio to the width of the picture. If it does not exist, the value of loc_focal_center_x is It is inferred to be the same as, and the value corresponds to the horizontal center of the picture. The value of loc_focal_center_x is between 0 and 2 24 It must be within the following range. Defines the vertical position of the focus within the picture, expressed as a ratio to the picture height. If it does not exist, the value of loc_focal_center_y is It is inferred to be the same as, and the value corresponds to the vertical center of the picture. The value of loc_focal_center_y is between 0 and 2 24It must be within the following range. loc_focal_length loc_focal_length defines the focal length of the lens, that is, the distance from the lens where light rays incident on the lens parallel to the optical axis converge. This is expressed in units of 0.1 millimeters. A value of 0 indicates that the focal length is not defined. If it does not exist, the value of loc_focal_length is inferred to be 0.

[0189] loc_rd_parameters_present_flag A value equal to 1 indicates that radial distortion parameters are present in the lens optical correction SEI message syntax. A value equal to 0 indicates that radial distortion parameters are not present in the lens optical correction SEI message syntax. loc_no_rd A value equal to 1 indicates that there is no radiative distortion. A value equal to 0 indicates that radiative distortion information is provided by the radiative distortion parameters of the lens optical correction SEI message. loc_rd_model_id Identifies the type of radiation distortion model specified in Table 1. The value of loc_rd_model_id must be within the range of 0 to 2. Values ​​greater than 2 are reserved for future use by ITUT | ISO / IEC and must not exist in bitstreams compliant with this version of the specification. The decoder must ignore lens optical correction SEI messages where loc_rd_model_id is greater than 2.

[0190] 표 5 - loc_rd_model_id

[0191]

[0192] When loc_rd_model_id is equal to 0, the polynomial (Poly3) model is specified as follows, and

[0193]

[0194] Here, r d represents the distorted radius (normalized distance to the image center), and ru represents the corrected non-distortion radius, and k1 is the distortion coefficient.

[0195] When loc_rd_model_id is equal to 1, the polynomial 4th-order (Poly4) model is specified as follows, and

[0196]

[0197] Here, r d represents the distorted radius (normalized distance to the image center), and r u represents the corrected non-distortion radius, and k1, k2, k3, and k4 are distortion coefficients.

[0198] A lens having radiation distortion information expressed by a three-parameter formula of the form, By setting it to this, it can be signaled as a Poly4 parameter.

[0199] When loc_rd_model_id is equal to 2, the 5th-order polynomial (Poly5) model is specified as follows, and

[0200]

[0201] Here, rd represents the distorted radius (normalized distance to the center of the image), ru represents the corrected undistorted radius, and k1 and k2 are distortion coefficients.

[0202] represents the k1 parameter value of the Poly3 radiation distortion model of Equation (X-1). If it does not exist, the value of loc_rd_poly3_k1 is inferred to be 0. represents the k1 parameter value of the Poly4 radiation distortion model in Equation (X-2). If it does not exist, the value of loc_rd_poly4_k1 is inferred to be 0. represents the k2 parameter value of the Poly4 radiation distortion model in Equation (X-2). If it does not exist, the value of loc_rd_poly4_k2 is inferred to be 0. represents the k3 parameter value of the Poly4 radiation distortion model in Equation (X-2). If it does not exist, the value of loc_rd_poly4_k3 is inferred to be 0. represents the k4 parameter value of the Poly4 radiation distortion model in Equation (X-2). If it does not exist, the value of loc_rd_poly4_k4 is inferred to be 0. represents the k1 parameter value of the Poly5 radiation distortion model of Equation (X-2). If it does not exist, the value of loc_rd_poly5_k1 is inferred to be 0. represents the k2 parameter value of the Poly5 radiation distortion model of Equation (X-2). If it does not exist, the value of loc_rd_poly5_k2 is inferred to be 0. loc_tca_parameters_present_flag That is equal to 1 indicates that the transverse chromatic aberration parameter exists within the lens optical correction SEI message syntax.

[0203] loc_tca_parameters_parameters_present_flag being equal to 0 indicates that the transverse chromatic aberration parameter is not present in the lens optical correction SEI message syntax. loc_no_tca being equal to 1 indicates that there is no transverse chromatic aberration. loc_no_tca That is equal to 0 indicates that lateral chromatic aberration information is provided by the lateral chromatic aberration parameter within the lens optical correction SEI message. loc_tca_model_id Identifies the type of radiation distortion model specified in Table 2. The value of loc_tca_model_id must be within the range of 0 to 1. Values ​​greater than 1 are reserved for future use by ITUT | ISO / IEC and must not exist in bitstreams compliant with this version of the specification. The decoder must ignore lens optical correction SEI messages where loc_tca_model_id is greater than 1.

[0204] 표 6 - loc_tca_model_id

[0205]

[0206] When loc_tca_model_id is equal to 0, the linear model is specified as follows, and

[0207]

[0208] Here, r d,R and r d,B and represent the distorted radii (normalized distance to the image center) of the red and blue components, respectively, and r u,R and r u,B and represent the corrected undistorted radii of the red and blue components, respectively, and k R and k B are the transverse chromatic aberration coefficients of the red and blue components, respectively.

[0209] Lateral chromatic aberration is corrected in a linear model for red and blue components based on the green component.

[0210] When loc_tca_model_id is equal to 1, the polynomial (Poly3) model is specified as follows, and

[0211]

[0212] Here, r d,R and r d,B and represent the distorted radii (normalized distance to the image center) of the red and blue components, respectively, and r u,R and r u,B represents the corrected non-distortion radius of the red and blue components, respectively, and {b R , c R , v R} and {b B , c B , v B} are the transverse chromatic aberration coefficients of the red and blue components, respectively.

[0213] Lateral chromatic aberration is corrected in the Poly3 model for the red and blue components based on the green component.

[0214] represents the kR parameter value of the linear transverse chromatic aberration model in Equation (X-3). If it does not exist, the value of loc_tca_kr is It is inferred to be something like this. k of the linear transverse chromatic aberration model in Equation (X-4). B Represents the parameter value. If it does not exist, the value of loc_tca_kr is It is inferred to be something like this. b from the Poly3 transverse chromatic aberration model of Equation (X-5) B Represents the value of the parameter. If it does not exist, the value of loc_tca_br is inferred to be 0. c of the Poly3 transverse chromatic aberration model of Equation (X-5). R Represents the parameter value. If it does not exist, the value of loc_tca_cr is inferred to be 0. v of the Poly3 transverse chromatic aberration model of Equation (X-5) R Represents the parameter value. If it does not exist, the value of loc_tca_vr is It is inferred to be something like this. is b of the Poly3 transverse chromatic aberration model of Equation (X-6). B Represents the parameter value. If it does not exist, the value of loc_tca_bb is inferred to be 0. c of the Poly3 transverse chromatic aberration model of Equation (X-6). B Represents the parameter value. If it does not exist, the value of loc_tca_cb is inferred to be 0.

[0215] v of the Poly3 transverse chromatic aberration model in Equation (X-6). B Indicates the value of the parameter. If it does not exist, the value of loc_tca_vb is It is inferred to be something like this. loc_vignetting_parameters_present_flag A value equal to 1 indicates that vignetting parameters are present in the lens optical correction SEI message syntax. A value equal to 0 indicates that vignetting parameters are not present in the lens optical correction SEI message syntax. loc_no_vignetting This being equal to 1 indicates that there is vignetting. loc_no_vignetting being equal to 0 indicates that vignetting information is provided by the vignetting parameter of the lens optical correction SEI message. loc_vignetting_model_id Identifies the type of vignetting model specified in Table 1. The value of loc_vignetting_model_id must be equal to 0. Values ​​greater than 0 are reserved for future use by ITUT | ISO / IEC and must not exist in bitstreams compliant with this version of the specification. The decoder must ignore lens optical correction SEI messages where loc_vignetting_model_id is greater than 0.

[0216] 표 7 - loc_vignetting_model_id

[0217]

[0218] When loc_vignetting_model_id is equal to 0, the 6th-order polynomial (Poly6) model is specified as follows, and

[0219]

[0220] Here, c d represents a distorted color, and c s represents undistorted color, and r u represents the radius, and k1, k2, and k3 are vignetting distortion coefficients.

[0221] The same vignetting parameters are used for red, green, and blue.

[0222] represents the k1 parameter value of the Poly6 vignette model in Equation (X-2). If it does not exist, the value of loc_vignetting_k1 is inferred to be 0. represents the value of the k2 parameter of the Poly6 vignette model in Equation (X-2). If it does not exist, the value of loc_vignetting_k2 is inferred to be 0. represents the k3 parameter value of the Poly6 vignette model in Equation (X-2). If it does not exist, the value of loc_vignetting_k3 is inferred to be 0.

[0223] Accordingly, according to the present embodiment, the encoder can read a configuration file containing Loc SEI parameters and insert a Loc SEI message into a bitstream. The decoder can parse the Loc SEI message from the bitstream, apply radial distortion and / or lateral chromatic aberration correction to the decoded video, and output the corrected video. Further examples are described below.

[0224] According to one embodiment, optical correction processing of the video is implemented in the LensOpticalCorrection class defined in LensOpticalCorrection.h / cpp as part of the CommonLib library. The LensOpticalCorrection::process function handles both radial distortion and TCA correction. For TCA correction, conversion to RGB (BT.709) is performed before applying the correction. The result is then converted back to YUV. This is merely for illustrative purposes. A bilinear filter is used for radial distortion correction, but here too, the purpose is for illustrative purposes only, and implementations using these correction parameters may use more advanced filters.

[0225] FIG. 10 illustrates an example (1000) of parameters in a configuration file as a sample configuration of a lens optical correction SEI according to an embodiment. On the encoder side, a lens optical correction configuration file is added by adding the -c option: EncoderApp -c cfg / encoder_lowdelay_vtm.cfg -i src.yuv -c cfg / sei_vui / lens_optical_correction.cfg. And outputting the lens-corrected version is performed, for example, as DecoderApp -b str.bin -o output.yuv --SEILocFilename='loc_output.yuv'.

[0226] FIG. 11 illustrates an example (1100) of radiation distortion correction implemented according to an embodiment of the present specification. In Example A of FIG. 11, radiation distortion correction was applied as follows using a Poly5 radiation distortion model signaled in a Loc SEI message to obtain an output corrected video.

[0227]

[0228] In Example B of Fig. 11, the following corrected output video was obtained by applying radiation distortion correction using Poly5 inserted into the Loc SEI message by calculating the parameters below.

[0229]

[0230] A decoder that interprets parameters and applies them to the original video generated a corrected version as exemplified in Fig. 11.

[0231] FIG. 12 illustrates an example (1200) in which transverse chromatic aberration (TCA) correction is implemented according to an embodiment of the present specification. With respect to TCA, as illustrated in FIG. 12, a shot with visible chromatic aberration was selected, as can be understood from the above description of chromatic aberration relating to, for example, FIG. 8a and FIG. 8b. And to correct the chromatic aberration, a polynomial of the third order (Poly3) method was used as follows.

[0232]

[0233] The result is illustrated in Fig. 12 with two enlarged visible areas (the original version on the left and the corrected version on the right).

[0234] Techniques for optical correction SEI messages for video coding as described above may be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media, or by one or more specially configured hardware processors. For example, FIG. 13 illustrates a computer system (1300) suitable for implementing a specific embodiment of the disclosed subject matter.

[0235] Accordingly, according to embodiments of the present specification, a method, apparatus, and computer system for video encoding such video data and SEI messages are provided, which also enables decoding by a decoder, and includes the step of the decoder receiving metadata associated with at least one picture representing an optical correction, wherein the metadata includes at least one parameter required for at least one optical correction model.

[0236] Computer software may be coded using any suitable machine code or computer language and may undergo assembly, compilation, linking, or similar mechanisms to generate code containing instructions that can be executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or through an interpreter, microcode execution, etc.

[0237] The instruction can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0238] The components of the computer system (1300) illustrated in FIG. 13 are exemplary and are not intended to imply any limitations regarding the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. Furthermore, the configuration of the components should not be interpreted as having any dependency or requirement related to any single component or combination of components illustrated in the exemplary embodiments of the computer system (1300).

[0239] The computer system (1300) may include a specific human interface input device. Such a human interface input device may respond to input by one or more human users through, for example, tactile input (e.g., key input, swipe, data glove motion), audio input (e.g., voice, clap), visual input (e.g., gesture), and olfactory input (not shown). The human interface device may also be used to capture specific media that are not necessarily directly related to conscious input by a human, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., 2D video, 3D video including stereoscopic video).

[0240] The input human interface device may include one or more of a keyboard (1301), a mouse (1302), a trackpad (1303), a touch screen (1310), a joystick (1305), a microphone (1306), a scanner (1308), and a camera (1307) (only one of each shown).

[0241] The computer system (1300) may also include specific human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (1310) or joystick (1305), but there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (1309), headphones (not shown)), visual output devices (e.g., screens (1310) including CRT screens, LCD screens, plasma screens, and OLED screens, each having or not having touchscreen input and tactile feedback functions, some of which may provide two-dimensional visual output or output exceeding three dimensions through means such as stereoscopic output; virtual reality glasses (not shown), holographic displays and smoke tanks (not shown)), and printers (not shown).

[0242] The computer system (1300) may also include human-accessible storage devices and associated media, such as optical media including CD / DVD ROM / RW (1320) or similar media including CD / DVD (1311), thumb drives (1322), removable hard drives or solid-state drives (1323), legacy magnetic media (not shown) such as tapes and floppy disks, and specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles.

[0243] Furthermore, a person skilled in the art should understand that the term "computer-readable medium," as used in connection with the subject matter currently disclosed, does not include a transmission medium, a carrier wave, or other transient signals.

[0244] The computer system (1300) may also include an interface (1399) to one or more communication networks (1398). The network (1398) may be, for example, wireless, wired, or optical. The network (1398) may additionally be local, wide-area, metropolitan, automotive and industrial, real-time, latency-tolerant, etc. Examples of the network (1398) include a local area network such as Ethernet, a wireless LAN, a cellular network including GSM, 3G, 4G, 5G, LTE, etc., a TV wired or wireless wide-area digital network including cable TV, satellite TV, and terrestrial broadcast TV, and an automotive and industrial network including CANBus. A specific network (1398) typically requires an external network interface adapter (e.g., a USB port of the computer system (1300)) attached to a specific general-purpose data port or peripheral bus (1350 and 1351); Other networks are typically integrated into the core of the computer system (1300) by connection to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Regardless of which of these networks (1398) is used, the computer system (1300) can communicate with other entities. Such communication may be receive-only unidirectional (e.g., broadcast TV), transmit-only unidirectional (e.g., CANbus for specific CANbus devices), or bidirectional, and may be to other computer systems using, for example, local or wide-area digital networks. As described above, specific protocols and protocol stacks may be used on each of such networks and network interfaces.

[0245] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (1340) of the computer system (1300).

[0246] The core (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), graphics adapters (1317), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (1343), hardware accelerators (1344) for specific tasks, etc. These devices may be connected via a system bus (1348), along with internal mass storage (1347), such as read-only memory (ROM) (1345), random access memory (1346), internal user-inaccessible hard drives, SSDs, etc. In some computer systems, the system bus (1348) may be accessed in the form of one or more physical plugs to enable expansion through additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1348) or via a peripheral bus (1351). Architectures of peripheral buses include PCI, USB, etc.

[0247] The CPU (1341), GPU (1342), FPGA (1343), and accelerator (1344) can be combined to execute specific instructions that can construct the aforementioned computer code. The computer code may be stored in ROM (1345) or RAM (1346). While temporary data may also be stored in RAM (1346), permanent data may be stored, for example, in internal mass storage (1347). High-speed storage and retrieval of any of the memory devices may be made possible through the use of a cache memory that may be closely associated with one or more CPUs (1341), GPUs (1342), mass storage (1347), ROM (1345), RAM (1346), etc.

[0248] Computer code for performing various computer-implemented operations may be recorded on a computer-readable medium. The medium and the computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the field of computer software.

[0249] As an example rather than a limitation, an architecture corresponding to a computer system (1300), in particular a core (1340), may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software implemented on one or more types of computer-readable media. Such computer-readable media may be a medium associated with user-accessible mass storage as described above, and may also be a specific storage device having the non-transient characteristics of the core (1340), for example, internal mass storage (1347) or ROM (1345). Software implementing various embodiments of the present disclosure may be stored in such a device and executed by the core (1340). The computer-readable media may include one or more memory devices or chips as needed. Software may enable a core (1340), and specifically a processor (CPU, GPU, FPGA, etc.) within it, to execute a specific process or a specific part of a specific process described herein, including defining data structures stored in RAM (1346) and modifying such data structures according to a process defined by the software. Additionally or alternatively, the computer system may provide a function as a result of logic hardwired to or otherwise implemented in a circuit (e.g., an accelerator (1344)) that can operate on behalf of or with the software to execute a specific process or a specific part of a specific process described herein. References to software may include logic, and where appropriate, vice versa. References to computer-readable media may include a circuit that stores software for execution (e.g., an integrated circuit (IC)), a circuit that implements logic for execution, or, where appropriate, all of these. The present disclosure includes any suitable combination of hardware and software.

[0250] Although the present disclosure describes various exemplary embodiments, modifications, substitutions, and various equivalent substitutions fall within the scope of the present disclosure. Accordingly, those skilled in the art will understand that numerous systems and methods can be devised that embody the principles of the present disclosure and fall within the spirit and scope thereof, even if they are not expressly illustrated or described in this specification.

Claims

Claim 1 A method for video decoding in a decoder, the method comprising: receiving a video bitstream including at least one picture of a video and a supplementary enhancement information (SEI) message associated with said at least one picture; obtaining metadata from said SEI message indicating one or more parameters of an optical correction to be applied to said at least one picture; and decoding said picture and applying said optical correction to said picture based on said one or more parameters of said optical correction. Claim 2 A method according to claim 1, wherein the SEI message includes a lens optical correction (loc) SEI message syntax of "lens_optical_correction". Claim 3 A method according to paragraph 2, wherein the metadata indicates that one or more parameters of the optical correction are applied based on any of the radiative distortion, chromatic aberration, and vignetting of the picture. Claim 4 A method according to paragraph 3, wherein the loc SEI message syntax further comprises a plurality of flags indicating whether the optical correction includes correction for radiative distortion, chromatic aberration, and vignetting, respectively. Claim 5 A method according to claim 1, wherein the SEI message indicates a first focal center of the picture along a first axis, a second focal center of the picture along a second axis, and a focal length. Claim 6 In paragraph 5, the above SEI message further indicates a lens model, a method. Claim 7 A method according to claim 1, wherein the SEI message further represents a set of parameters of at least one model including the one or more parameters, and the optical correction is applied to at least one picture by the at least one model. Claim 8 A video encoding method comprising: receiving a video including at least one picture; determining an optical correction to be applied to the at least one picture; and encoding the at least one picture and a supplementary enhancement information (SEI) message associated with the at least one picture, wherein the SEI message represents one or more parameters of the optical correction to be applied to the at least one picture. Claim 9 In claim 8, the method wherein the SEI message includes the lens optical correction (loc) SEI message syntax of "lens_optical_correction". Claim 10 In claim 9, the metadata indicates that one or more parameters of the optical correction are applied based on any of the radiative distortion, chromatic aberration, and vignetting of the picture. Claim 11 A method according to claim 10, wherein the above loc SEI message syntax further comprises a plurality of flags indicating whether the optical correction includes correction for radial distortion, chromatic aberration, and vignetting, respectively. Claim 12 In paragraph 8, the method wherein the SEI message indicates a first focal center of the picture along a first axis, a second focal center of the picture along a second axis, and a focal length. Claim 13 In Clause 12, the above SEI message further indicates a lens model, a method. Claim 14 In claim 8, the SEI message further represents a set of parameters of at least one model including one or more parameters, and the optical correction is applied to at least one picture by the at least one model. Claim 15 A method for processing visual media data, wherein the method comprises the step of performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the format rule instructs determining one or more parameters of optical correction to be applied to at least one picture of a video, and determining said one or more parameters based on a supplementary enhancement information (SEI) message associated with said at least one picture. Claim 16 In paragraph 15, the method wherein the above SEI message includes the lens optical correction (loc) SEI message syntax of "lens_optical_correction". Claim 17 In paragraph 16, the metadata indicates that one or more parameters of the optical correction are applied based on any of the radiative distortion, chromatic aberration, and vignetting of the picture. Claim 18 In claim 17, the above loc SEI message syntax further comprises a plurality of flags indicating whether the optical correction includes correction for radiative distortion, chromatic aberration, and vignetting, respectively. Claim 19 In paragraph 15, the above SEI message indicates a first focal center of the picture along a first axis, a second focal center of the picture along a second axis, and a focal length, and the above SEI message further indicates a lens model, method. Claim 20 In paragraph 15, the SEI message further represents a set of parameters of at least one model including the one or more parameters, and the optical correction is applied to at least one picture by the at least one model.