Data processing method and apparatus, and system

Abstract

Disclosed are a data processing method and apparatus, and a system, relating to the technical field of video processing or image processing. An encoder acquires base layer data, enhancement layer data, and metadata of a first image, and encodes the base layer data, the enhancement layer data, and the metadata to obtain a code stream. A decoder receives and parses the code stream to obtain the base layer data, the enhancement layer data, and the metadata of the first image. The decoder determines and converts target data (i.e., the base layer data, the enhancement layer data or an image obtained by combining the base layer data and the enhancement layer data) on the basis of conversion information carried in the metadata. Therefore, the decoder determines and converts the target data on the basis of the conversion information carried in the metadata, so that the conversion of the target data matches the target data; and the decoder converts the target data using the conversion information matching the target data, so that a dynamic range of an image obtained by said conversion can better adapt to a dynamic range supported by a display device, and a display effect is also improved.

Description

Data processing method, device and system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 7, 2023, with application number 202311289650.8 and application name “Encoding method and related equipment”, and the Chinese patent application filed with the State Intellectual Property Office on January 30, 2024, with application number 202410139746.4 and application name “A data processing method, device and system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of video or image processing technology, and in particular to a data processing method, device, and system. Background Art

[0003] In the field of video or image processing, dynamic range refers to the ratio of the maximum grayscale value (or brightness value) of an image pixel to the minimum grayscale value.

[0004] After an electronic device captures a video or image, it can display the video or image on the electronic device or transmit the video or image to another display device for display on the other display device. Different display devices can display different dynamic ranges of pixels. When displaying a video or image on a display device, the dynamic range of the captured video or image may be different from the dynamic range supported by the display device. Therefore, the dynamic range of the captured video or image needs to be adjusted to the dynamic range supported by the display device so that the video or image can be successfully displayed on the display device.

[0005] Electronic devices can convert the dynamic range of a video or image based on the dynamic range supported by the display device, so that the converted dynamic range of the video or image matches the dynamic range supported by the display device. This conversion of the dynamic range of a video or image based on the dynamic range supported by the display device affects encoding efficiency.

[0006] Summary of the Invention

[0007] The present application provides a data processing method, device, and system for solving the problem that the dynamic range of a video or image is converted by an electronic device based on the dynamic range supported by the display device, which affects the coding efficiency.

[0008] In a first aspect, an embodiment of the present application provides a data processing method. The method is executed by a decoder, and the decoder determines target data based on conversion information carried in metadata. The metadata is metadata of a first image. The target data includes at least one of base layer data of the first image and enhanced layer data of the first image. For example, the target data is base layer data of the first image, or the target data is enhanced layer data of the first image, or the target data is base layer data of the first image and enhanced layer data of the first image. Or the target data includes a second image. The second image is obtained based on the base layer data of the first image and the enhanced layer data of the first image. The decoder converts the target data based on the conversion information.

[0009] In this application, the decoder determines and converts the target data based on the conversion information carried in the metadata, so that the conversion of the target data matches the target data. The decoder uses the conversion information that matches the target data to convert the target data, so that the dynamic range of the converted image can not only better adapt to the dynamic range supported by the display device, but also improve the display effect.

[0010] In one possible implementation, the metadata can be pre-set or carried in the bitstream. If the metadata is carried in the bitstream, the decoder retrieves the metadata from the bitstream. In this way, the bitstream carries the metadata for the first image, allowing the decoder to quickly and accurately retrieve the metadata, improving conversion efficiency.

[0011] In another possible implementation, the conversion information includes conversion location information and conversion scheme information. The conversion location information indicates the location where the conversion occurs, and the conversion scheme information indicates the conversion scheme used when converting the target data. The conversion occurs before or after the second image is obtained. In this way, the decoder can determine the conversion scheme to be executed based on the conversion location information, allowing it to quickly obtain a conversion scheme that matches the target data, thereby improving decoding efficiency.

[0012] In another possible implementation, when the conversion occurs before the second image is obtained, the conversion information also includes conversion target information, where the conversion target information indicates the conversion target for which the conversion scheme is to be performed. The conversion target includes base layer data, enhancement layer data, or both base layer data and enhancement layer data. In this manner, the decoder determines a conversion scheme that matches the conversion target information based on the conversion target information, thereby improving the conversion effect.

[0013] In another possible implementation, when the conversion target includes base layer data, the conversion scheme information includes a base layer data conversion scheme, where the base layer data conversion scheme indicates that at least one of color gamut conversion, chroma format conversion, color compensation, and color mode conversion should be performed on the base layer data. In this manner, the decoder performs at least one of the color gamut conversion, chroma format conversion, color compensation, and color mode conversion on the base layer data according to the base layer data conversion scheme, so that the converted image is better adapted to the display device and has an improved display quality.

[0014] In another possible implementation, the base layer data conversion scheme is further used to indicate conversion parameters used when converting the base layer data.

[0015] In another possible implementation, the conversion parameters include at least one of a compensation matrix, a conversion matrix, a conversion function, and a compensation value. In this way, the decoder can quickly and accurately use the conversion parameters that match the conversion process and convert the base layer data using the conversion parameters that match the conversion process, thereby improving the conversion effect.

[0016] In another possible implementation, when the conversion target includes enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme. The enhancement layer data conversion scheme indicates that at least one of color mode conversion, chroma format conversion, and channel conversion should be performed on the enhancement layer data. The channel conversion is used to restore the channels of the enhancement layer data to their original channels. In this manner, the decoder performs at least one of color mode conversion, chroma format conversion, and channel conversion on the base layer data according to the base layer data conversion scheme, so that the converted image is better adapted to the display device and improves the display quality.

[0017] In another possible implementation, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the enhancement layer data.

[0018] In another possible implementation, the conversion parameter includes at least one of a conversion matrix, a conversion function, and a channel conversion function.

[0019] In a possible scenario, the conversion information includes channel merging information. The channel merging information is used to indicate a combination scheme of channels corresponding to the color mode of the base layer data and channels corresponding to the color mode of the enhancement layer data.

[0020] In another possible implementation, the color modes of the base layer data and the enhancement layer data may include multiple different combinations, several of which are given below.

[0021] Mode 1: The color mode of at least one of the base layer data and the enhancement layer data is a red, green, and blue (RGB) mode or a grayscale mode.

[0022] Mode 2: The color mode of the base layer data is RGB mode, and the color mode of the enhancement layer data is any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode.

[0023] Mode 3: The color mode of the base layer data is any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode; the color mode of the enhancement layer data is RGB mode.

[0024] Mode 4: The color mode of the base layer data and the color mode of the enhancement layer data are both any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode.

[0025] In one possible scenario, according to the different color modes of the enhancement layer data of the base layer data, the channel merging manner of the base layer data and the enhancement layer data indicated by the channel merging information is also different. Several possible manners are given below.

[0026] Method 1: When the color modes of the basic layer data and the enhanced layer data are both RGB modes, the channel merging information is used to indicate: the R channel of the basic layer data and the R channel of the enhanced layer data are merged, the G channel of the basic layer data and the G channel of the enhanced layer data are merged, and the B channel of the basic layer data and the B channel of the enhanced layer data are merged.

[0027] Method 2: When the color mode of the basic layer data is the RGB mode and the color mode of the enhanced layer data is the grayscale mode, the channel merging information is used to indicate: the R channel of the basic layer data is merged with the grayscale value of the enhanced layer data, the G channel of the basic layer data is merged with the grayscale value of the enhanced layer data, and the B channel of the basic layer data is merged with the grayscale value of the enhanced layer data.

[0028] Method 3: When the color mode of the basic layer data is the grayscale mode and the color mode of the enhanced layer data is the RGB mode, the channel merging information is used to indicate: the R channel of the enhanced layer data is merged with the grayscale value of the basic layer data, the G channel of the enhanced layer data is merged with the grayscale value of the basic layer data, and the B channel of the enhanced layer data is merged with the grayscale value of the basic layer data.

[0029] Mode 4: When the color modes of the base layer data and the enhancement layer data are both grayscale modes, the channel merging information is used to indicate that the grayscale value of the base layer data and the grayscale value of the enhancement layer data are merged.

[0030] Method 5: When the color mode of the basic layer data and the color mode of the enhanced layer data are both any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0031] Method 6: When the color modes of the basic layer data and the enhanced layer data are any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, the channel merging information is used to indicate that the brightness channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0032] Method 7: When the color mode of the basic layer data is grayscale mode and the color mode of the enhanced layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, the channel merging information is used to indicate: the channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0033] Method 8: When the color mode of the basic layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, and the color mode of the enhanced layer data is the grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the channel of the enhanced layer data are merged.

[0034] In this way, the decoder merges the channels of the base layer data and the enhancement layer data using a channel merging scheme that matches the color modes of the base layer data and the enhancement layer data, thereby improving the display effect of the merged image.

[0035] In another possible implementation, the channel merging method may include but is not limited to: addition, multiplication, etc.

[0036] In another possible implementation, the color mode conversion includes: converting the color mode of the base layer data so that the converted color mode of the base layer data is the same as the color mode of the enhancement layer data; or converting the color mode of the enhancement layer data so that the converted color mode of the enhancement layer data is the same as the color mode of the base layer data.

[0037] In another possible implementation, when the conversion occurs after the second image is obtained, the conversion scheme information includes at least one of: color gamut conversion, chroma format conversion, color compensation, and color mode conversion.

[0038] In another possible implementation, the conversion solution information further includes conversion parameters, which include at least one of a compensation matrix, a conversion matrix, a conversion function, and a compensation value.

[0039] In another possible implementation manner, the conversion information includes at least one or more of the following manners 1 to 3.

[0040] Mode 1: color mode information of the base layer data of the first image.

[0041] Mode 2: color mode information of the enhancement layer data of the first image.

[0042] Mode 3: Color mode information corresponding to the base layer data of the first image and the enhancement layer data of the first image that needs to be transformed and processed.

[0043] In another possible implementation, when the image color space model of the base layer data is a first color mode, and the color space model when the base layer data is processed in combination with the enhancement layer data is a second color mode, and the first color mode and the second color mode are different, the converted and processed color mode information includes a color mode conversion method. In this manner, the decoder converts the image color space model of the base layer data and / or the image color space model of the enhancement layer data so that the converted image color space model of the base layer data or the enhancement layer data matches the color space model when the base layer data and the enhancement layer data are processed in combination, thereby enabling the base layer data and the enhancement layer data to be effectively merged and improving the display effect.

[0044] In another possible implementation, the first color mode and the second color mode are one of the following: RGB, YUV, YCbCr, Lab, HSV, and ICtCp.

[0045] In another possible implementation manner, the conversion information includes at least one or more of the following manners 1 to 3.

[0046] Mode 1: Chroma format information of the base layer data of the first image.

[0047] Mode 2: Chroma format information of the enhancement layer data of the first image.

[0048] Mode 3: the chroma format information corresponding to the base layer data of the first image and the enhancement layer data of the first image needs to be transformed and processed.

[0049] In another possible implementation, when the chroma format information of the base layer data of the first image is first chroma format information, the chroma format information when the base layer data and the enhancement layer data are processed together is second chroma format information, and the first chroma format information and the second chroma format information are different, the chroma format information that needs to be converted and processed includes chroma format conversion information. In this way, the decoder converts the chroma format of the base layer data and / or the chroma format of the enhancement layer data so that the converted chroma format of the base layer data or the enhancement layer data matches the chroma format when the base layer data and the enhancement layer data are processed together, so that the base layer data and the enhancement layer data can be effectively merged and the display effect can be improved.

[0050] In another possible implementation, the first chroma format information and the second chroma format information are one of the following: RGB, grayscale, YUV4:4:4, YUV4:2:2, or YUV4:2:0.

[0051] In another possible implementation, the first image and the second image are HDR or SDR images.

[0052] In a second aspect, embodiments of the present application provide an encoding method. The method is performed by an encoder, which determines, based on a first image acquired, base layer data of the first image, enhancement layer data of the first image, and metadata of the first image. The metadata carries conversion information. The encoder encodes the base layer data, enhancement layer data, and metadata to generate a bitstream.

[0053] Compared to an encoder directly encoding the first image to obtain a bitstream, in this application, the encoder obtains the first image's base layer data, enhancement layer data, and metadata based on the first image, and then encodes the base layer data, enhancement layer data, and metadata to obtain a bitstream. This way, the encoded bitstream contains more information about the first image, ensuring better image quality when decoded.

[0054] In one possible implementation, the conversion information includes conversion location information and conversion scheme information. The conversion location information indicates the location where the conversion occurs, and the conversion scheme information indicates the conversion scheme used when converting the target data. The conversion occurs before or after obtaining the second image. For details regarding the conversion information, conversion location information, conversion scheme information, and conversion scheme, please refer to the relevant content in the first aspect and will not be repeated here.

[0055] In a third aspect, an embodiment of the present application provides a method for processing a signal, the method comprising: obtaining first base layer data, first enhancement layer data, and first metadata. The first base layer data is base layer data of a first image, the first metadata is metadata of the first image, and the first enhancement layer data is metadata of the first image. Based on conversion information carried in the first metadata, target data is determined, where the target data is the first base layer data and / or the first enhancement layer data, or the target data is a second image, where the second image is determined based on the first base layer data and the first enhancement layer data. The target data is converted based on the conversion information.

[0056] The first metadata may be preset or carried in the bitstream.

[0057] In a possible implementation, the first image and the second image are HDR or SDR images.

[0058] In one possible embodiment, the conversion information is conversion indication information, and the conversion indication information includes conversion position information. When the conversion position is before determining the second image, the conversion indication information also includes conversion object information, and the conversion object information is used to indicate that the target data includes: the first basic layer data, or the first enhancement layer data, or the first basic layer data and the first enhancement layer data.

[0059] In a possible implementation, when the target data includes the first basic layer data, the conversion scheme information includes: a basic layer data conversion scheme. The basic layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the first basic layer data:

[0060] Color gamut conversion;

[0061] Color compensation; or

[0062] Color space conversion.

[0063] In a possible implementation manner, the base layer data conversion scheme is further used to indicate conversion parameters used when converting the first base layer data.

[0064] In a possible implementation manner, when the base layer data conversion scheme indicates to perform color compensation on the first base layer data, the conversion parameter includes any one of the following parameters:

[0065] Color compensation matrix;

[0066] a conversion function; or

[0067] Compensation value and compensation method.

[0068] In one possible implementation, the conversion information is conversion indication information. The conversion indication information includes conversion scheme information. When the target data includes the first enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme. The enhancement layer data conversion scheme is used to indicate that at least one of the following conversions be performed on the first enhancement layer data:

[0069] Color space conversion;

[0070] Channel conversion, where the channel conversion is used to restore the channels of the first enhancement layer data to the original channels.

[0071] In a possible implementation manner, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the first enhancement layer data.

[0072] In a possible implementation, when the target data is the second image, the conversion scheme information includes: an image conversion scheme, where the image conversion scheme is used to indicate performing at least one of the following conversions on the second image:

[0073] Color gamut conversion; or

[0074] Color compensation.

[0075] In a fourth aspect, an embodiment of the present application provides an encoding method, comprising: determining, based on a first acquired image, first base layer data, first enhancement layer data, and first metadata, wherein the first metadata carries conversion information; and encoding the first base layer data, the first enhancement layer data, and the first metadata to obtain a bitstream.

[0076] In one possible implementation, the conversion information is conversion indication information, which includes conversion location information and conversion scheme information, wherein the conversion location information is used to indicate the location where the conversion scheme occurs, and the conversion scheme information is used to indicate the conversion scheme. The first metadata may be preset or carried in the bitstream.

[0077] In a possible implementation, the conversion occurs before or after a new HDR image is synthesized based on the received code stream.

[0078] In a possible implementation, when the location where the conversion occurs is before synthesizing a new HDR image, the conversion indication information further includes conversion object information, where the conversion object information is used to indicate target data on which the conversion scheme needs to be executed, and the target data includes:

[0079] Base layer data; or

[0080] Enhancement layer data; or

[0081] The base layer data and the enhancement layer data.

[0082] In a possible implementation manner, when the target data includes the base layer data, the conversion scheme information includes: a base layer data conversion scheme, where the base layer data conversion scheme is used to indicate performing at least one of the following conversions on the base layer data:

[0083] Color gamut conversion;

[0084] Color compensation; or

[0085] Color space conversion.

[0086] In a possible implementation manner, the base layer data conversion scheme is further used to indicate conversion parameters used during conversion.

[0087] In a possible implementation manner, when the base layer data conversion scheme indicates to perform the color compensation, the conversion parameter includes any one of the following parameters:

[0088] Color compensation matrix;

[0089] a conversion function; or

[0090] Compensation value and compensation method.

[0091] In a possible implementation manner, when the target data includes the enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, where the enhancement layer data conversion scheme is used to indicate performing at least one of the following conversions on the enhancement layer data:

[0092] Color space conversion;

[0093] Channel conversion, where the channel conversion is used to restore the channels of the enhancement layer data to the original channels.

[0094] In a possible implementation manner, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the first enhancement layer data.

[0095] In a possible implementation, when the conversion scheme occurs after the new HDR image is synthesized, the conversion scheme information includes: an image conversion scheme, where the image conversion scheme is used to indicate that at least one of the following conversions is performed on the new HDR image:

[0096] Color gamut conversion; or

[0097] Color compensation.

[0098] In the embodiments of the present application, the color mode information of the base layer and / or the color mode information of the enhancement layer image / enhancement data and / or the color mode information of the conversion and processing of the base layer and the enhancement layer image / enhancement data can be preset or transmitted in the metadata, and the base layer image and / or the enhancement layer image / enhancement data can be converted based on this information, so that the conversion is more flexible, the metadata can achieve multi-format adaptation, and can be conducive to more efficient and reasonable encoding.

[0099] The embodiments of the present application can convert the base layer image and / or the enhancement layer image / enhancement data based on the chroma format information of the base layer and / or the chroma format information of the enhancement layer image / enhancement data, and / or the chroma format information of the conversion and processing of the base layer and the enhancement data, thereby achieving a higher degree of matching with the color gamut of different display devices, supporting color space conversion, and facilitating more efficient and reasonable encoding.

[0100] The embodiments of the present application can provide information on how to combine the color mode multi-channel of the basic layer with the color mode multi-channel or single channel of the enhancement layer image / enhancement data in preset or transmitted metadata, thereby making the conversion more flexible, and the metadata can achieve flexible adaptation in various ways.

[0101] In a fifth aspect, an embodiment of the present application provides a decoding device, which includes modules for executing the decoding method in the first aspect or any possible design of the first aspect.

[0102] In a sixth aspect, embodiments of the present application provide a coding device, wherein the device includes modules for executing the coding method in the second aspect or any possible design of the second aspect.

[0103] In a seventh aspect, an embodiment of the present application provides an electronic device. The electronic device includes at least one processor and a memory, the memory being configured to store a set of computer instructions; when the processor executes the set of computer instructions as an execution device in the first aspect or any possible implementation of the first aspect, the operation steps of the decoding method in the first aspect or any possible implementation of the first aspect are performed; or when the processor executes the set of computer instructions as an execution device in the second aspect or any possible implementation of the second aspect, the operation steps of the encoding method in any two possible implementations of the first aspect or the second aspect are performed.

[0104] In an eighth aspect, an embodiment of the present application provides a coding and decoding system. The system includes a first electronic device and a second electronic device. The first electronic device is configured to perform the decoding method of the first aspect or any possible design of the first aspect, and the second electronic device is configured to perform the encoding method of the first aspect or any possible design of the first aspect.

[0105] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer software instructions that, when executed in a computing device, cause the computing device to execute the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0106] In a tenth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed on a computer, the computer program product causes the computing device to execute the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0107] The beneficial effects of the third to eighth aspects above can be referred to the description of any implementation in the first or second aspects, and will not be repeated here. Based on the implementation provided by the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] FIG1 is a schematic diagram of dynamic range mapping;

[0109] Figure 2 is an image of the PQ photoelectric transfer function;

[0110] Figure 3 is an image of the HLG photoelectric transfer function;

[0111] Figure 4 is an image of the SLF photoelectric transfer function;

[0112] FIG5 is a schematic diagram of the end-to-end process of the HDR object provided by this application;

[0113] FIG6 is a schematic diagram of the end-to-end system architecture of the HDR object provided by this application;

[0114] FIG7 is a schematic diagram of an end-to-end image processing system provided by the present application;

[0115] FIG8 is a schematic diagram of a workflow of a streaming media system provided by this application;

[0116] FIG9 is a schematic diagram of a streaming media system architecture provided by this application;

[0117] FIG10 is a schematic block diagram of a coding and decoding system used in this application;

[0118] FIG11 is a block diagram of a content providing system provided by the present application;

[0119] FIG12 is a schematic diagram of a workflow of a terminal device provided by the present application;

[0120] FIG13A is a schematic diagram of the first image processing process provided by the present application;

[0121] FIG13B is a schematic diagram of the second image processing process provided by the present application;

[0122] FIG13C is a schematic diagram of a third image processing process provided by the present application;

[0123] FIG13D is a schematic diagram of a fourth image processing process provided by the present application;

[0124] FIG13E is a schematic diagram of the fifth image processing process provided by this application;

[0125] FIG13F is a schematic diagram of a sixth image processing process provided by the present application;

[0126] FIG14 is a flow chart of a data processing method provided by the present application;

[0127] FIG15 is a schematic diagram of the first conversion process provided by this application;

[0128] FIG16 is a schematic diagram of the second conversion process provided by this application;

[0129] FIG17 is a schematic diagram of the third conversion process provided by this application;

[0130] FIG18 is a schematic diagram of the fourth conversion process provided by this application;

[0131] FIG19 is a schematic structural diagram of an encoding device provided by the present application;

[0132] FIG20 is a schematic structural diagram of an encoding device provided by the present application;

[0133] FIG21 is a schematic structural diagram of a decoding device provided by the present application;

[0134] FIG22 is a schematic structural diagram of a decoding device provided in this application. DETAILED DESCRIPTION

[0135] In order to make the purpose, technical solutions and advantages of this application clearer, this application will be further described in detail below with reference to the accompanying drawings.

[0136] The term "at least one" used in this application refers to one or more than one, including one, two, three and more; "plurality" refers to two or more than two, including two, three and more.

[0137] The term "at least one" in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where any of a, b, c, ab, ac, bc, or abc can include a single a, a single b, a single c, and can also include multiple a, multiple b, and multiple c.

[0138] Furthermore, unless otherwise indicated, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish between multiple objects and are not used to define the order, timing, priority, or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria and do not indicate differences in the content, priority, or importance of the two criteria.

[0139] In addition, the terms "including" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules / units is not limited to the listed steps or modules and may also include steps or modules / units that are not listed.

[0140] In order to facilitate understanding of the embodiments of the present application, some concepts or terms involved in the embodiments of the present application are explained.

[0141] Color value: A numerical value corresponding to a specific image primary color component (such as R, G, B, or Y). Primary color values ​​can include linear primary color values ​​and nonlinear primary color values.

[0142] Linear primary color value: The linear primary color value is proportional to the light intensity. In one possible case, its value should be normalized to [0,1], referred to as E.

[0143] Nonlinear base color value: The nonlinear base color value is the normalized digital expression value of the image information, which is proportional to the digital coding value. In one possible case, its value should be normalized to [0,1], referred to as E′.

[0144] Digital code value: A digital representation of an image signal. Digital code value is used to represent nonlinear primary color values.

[0145] Electro-optical transfer function (EOTF): A conversion relationship from nonlinear primary color values ​​to linear primary color values.

[0146] Metadata: Data that describes the video source information carried in the video signal.

[0147] Dynamic metadata: metadata associated with each frame of an image that changes from frame to frame.

[0148] Static metadata: Metadata associated with an image sequence that remains unchanged within the image sequence.

[0149] Luminance signal (luma): represents the combination of nonlinear primary color signals, symbolized by Y'.

[0150] Luminance mapping: The luminance of the source image is mapped to the luminance of the target system.

[0151] Display adaptation: Processing of the video signal to adapt it to the display characteristics of the target display.

[0152] Source picture: The image input to the HDR pre-processing stage.

[0153] Mastering display: A reference display used when editing and producing video signals to determine the effects of video editing and production.

[0154] Linear scene light signal: In HDR video technology, the HDR video signal with content as scene light refers to the scene light captured by the camera / camera sensor, which is generally a relative value; the linear scene light signal is obtained after HLG encoding to obtain the HLG signal, which is a scene light signal and is nonlinear; the scene light signal generally needs to be converted into a display light signal through an optical-optical transfer function (OOTF) curve and displayed on a display device.

[0155] Linear display light signal: In HDR video technology, the HDR video signal with content as display light refers to the display light emitted by the display device, generally expressed as an absolute value in nits. The linear display light signal is PQ-encoded to obtain a PQ signal, which is a nonlinear display light signal. The display light signal is generally displayed on the display device according to its absolute brightness.

[0156] Optical-to-optical conversion factor (OOTF): A curve used in video technology to convert one optical signal into another.

[0157] Luma-chroma-chroma (LCC), luminance and chroma separate the three components of the video signal.

[0158] Dynamic range: In most fields, dynamic range is used to express the ratio of the maximum value to the minimum value of a variable. In the field of digital imaging, dynamic range is the ratio of the maximum grayscale value to the minimum grayscale value of the image pixel. Grayscale value can also be understood as brightness value, with units of nits or candela per square meter (cd / m 2 ), 1nit=1cd / m 2 .

[0159] Generally, the dynamic range of nature is relatively large, and the brightness of the night scene under the starry sky is about 10 -3 cd / m 2 , the sun itself is as bright as 10 9 cd / m 2 , so the dynamic range reaches 10 9 / 10 -3 =10 12 In real natural scenes, since the brightness of the sun and the brightness of the stars are not obtained at the same time, the dynamic range of the real natural environment scene may not reach 10 12 For real natural environment scenes, the dynamic range is generally in the order of 10 -3 to 10 6 within the range.

[0160] It is understandable that dynamic range is divided into high dynamic range (HDR) or low dynamic range (SDR). Low dynamic range can also be called standard dynamic range. -3 to 10 6Images with a brightness range of 0 to 255 are called high dynamic range images, while images with a brightness range of 0 to 255 are called low dynamic range images. In most current color digital images, each pixel in the R, G, and B channels is represented by one byte (i.e., 8 bits). In other words, the brightness range of each channel's pixels is 0 to 255, which is the standard dynamic range of the image. The imaging process of a digital camera is essentially a mapping of the real-world dynamic range to the dynamic range supported by the display device. This is often a nonlinear process, as shown in Figure 1, which is a schematic diagram of dynamic range mapping.

[0161] The optical-electrotransfer function (OETF) represents the conversion relationship between linear and nonlinear signals for image pixels. There are three types of OETFs: Perceptual Quantizer (PQ), Hybrid Log-Gamma (HLG), and Scene Luminance Fidelity (SLF). These three OETFs are specified by the Audio Video Coding Standard (AVS).

[0162] (1) PQ photoelectric transfer function

[0163] The PQ photoelectric transfer function (PQPTF) is a perceptual quantization PQPTF proposed based on the brightness perception model of the human eye. Figure 2 shows an image of the PQPTF. Computing devices can use the PQPTF to represent the conversion relationship between the linear signal value of an image pixel and the nonlinear signal value in the PQ domain. The PQPTF can be expressed as formula (1).

[0164] Where L represents the linear signal value, which is normalized to [0, 1]. L' represents the nonlinear signal value, which is in the range of [0, 1]. m1, m2, c1, c2, and c3 are the PQ photoelectric transfer coefficients, and m1, m2, c1, c2, and c3 satisfy the following relationship:

[0165] Taking the image color model as the three primary color light mode (red, green, blue, RGB) as an example, the conversion process of the RGB image using the PQ photoelectric transfer function can be expressed by formula (2).

[0166] (2) HLG photoelectric transfer function

[0167] The HLG photoelectric transfer function is an improvement on the traditional gamma curve. Figure 3 shows an image of the HLG photoelectric transfer function. As shown in Figure 3, the HLG photoelectric transfer function applies the traditional gamma curve in the low range and supplements the log curve in the high range. The HLG photoelectric transfer function represents the conversion relationship between the linear signal value of the image pixel and the nonlinear signal value in the HLG domain. The HLG photoelectric transfer function can be expressed as Equation (3).

[0168] Wherein, L represents the linear signal value, and its value range is [0, 12]; L' represents the nonlinear signal value, and its value range is [0, 1]; a, b, and c are HLG photoelectric transfer coefficients, a=0.17883277, b=0.28466892, and c=0.55991073.

[0169] (3)SLF Photoelectric Transfer Function

[0170] The SLF photoelectric transfer function (SLF) is an optimal curve obtained based on the HDR scene brightness distribution, while satisfying the optical characteristics of the human eye. Figure 4 shows an image of the SLF photoelectric transfer function. The SLF photoelectric transfer curve represents the conversion relationship between the linear signal value of an image pixel and the nonlinear signal value in the SLF domain. The conversion relationship between the linear signal value of an image pixel and the nonlinear signal value in the SLF domain can be expressed as Equation (4).

[0171] The SLF photoelectric transfer function can be expressed as formula (5).

[0172] Where L represents the linear signal value, which is normalized to [0, 1]. L' represents the nonlinear signal value, which ranges from [0, 1]. p, m, a, and b are the SLF photoelectric transfer coefficients, where p = 2.3, m = 0.14, a = 1.12762, and b = -0.12762.

[0173] Linear space: In this application, linear space refers to the space where linear light signals are located.

[0174] Nonlinear space: In this application, nonlinear space refers to the space resulting from the conversion of linear light signals using nonlinear curves. Commonly used nonlinear curves for HDR include the PQ EOTF-1 curve and the HLG OETF curve, while common nonlinear curves for SDR include the gamma curve. It is generally believed that linear light signals encoded using these nonlinear curves are visually linear relative to the human eye. It should be understood that nonlinear space can be considered a visually linear space.

[0175] Gamma correction: Gamma correction is a method of nonlinearly editing the tonal range of an image. It detects dark and light portions of the image signal and increases their ratio, thereby improving image contrast. The photoelectric conversion characteristics of current display screens, photographic film, and many electronic cameras can be nonlinear. The relationship between the output and input of these nonlinear components can be expressed as a power function: output = (input)γ.

[0176] The nonlinear conversion of the color values ​​output by the device is due to the nonlinear nature of the human visual system. Humans perceive visual stimuli through comparison. External stimuli are intensified at a certain rate, resulting in a uniform increase in the stimulus. Therefore, physical quantities that increase in geometric progression appear uniform to human perception. To display input colors in accordance with human visual principles, the aforementioned nonlinear conversion using the power function is necessary, converting linear color values ​​to nonlinear ones. The gamma value, γ, can be determined based on the photoelectric conversion curve in the color space.

[0177] Color space: Color can be the different perceptions of the eye for light of different frequencies, or it can represent objectively existing light of different frequencies. Color space is a color range defined by a coordinate system established by people to represent colors. The color gamut mode and color model together define a color space. Among them, the color model is an abstract mathematical model that represents color with a set of color components. The color model can include, for example, the three primary color light mode (red, green, blue, RGB) and the four-color printing mode (cyan, magenta, yellow key plate, CMYK). The color gamut refers to the sum of the colors that a system can produce. For example, Adobe RGB and sRGB are two different color spaces based on the RGB model. In this application, the color model can also be called a color mode.

[0178] RGB space refers to a space that uses the brightness of the three primary colors (red, green, and blue) to quantitatively represent video or image signals. YCC space is a color space that separates luminance and chrominance. The three components of a YCC video or image signal represent luminance, chrominance, and chrominance, respectively. Common YCC space video signals include YUV, YCbCr, and ICtCp. In this application, "luminance and chrominance separation" can be referred to as "luminance-color separation," while "luminance and chrominance not separated" can be referred to as "luminance-color not separated."

[0179] Each device, such as a monitor or printer, has its own color space and can only produce colors within its gamut. When you move an image from one device to another, the colors may change on different devices because each device converts and displays RGB or CMYK according to its own color space.

[0180] To obtain images with a higher dynamic range, the image bit width is usually greater than or equal to 10 bits (bit, b). Common coding standards that support HDR objects include H.266, H.265, and the High Efficiency Image Format (HEIF). The Joint Photographic Experts Group (JPEG) and H.264 only support 8-bit encoding and cannot well support HDR video and HDR images.

[0181] During the distribution or transmission process, HDR images or videos may need to have their dynamic range converted to a range supported by a display device (such as a TV or iPad). This allows the converted HDR image or video to match the dynamic range of the display device so that it can be displayed on the display device. The following two solutions are commonly used to convert the dynamic range of HDR images or videos.

[0182] (1) Single-layer transcoding solution.

[0183] HDR images or HDR videos may need to be transcoded during distribution or transmission. The current HDR transcoding solution is to directly transcode HDR images or HDR videos. After decoding, the HDR image or HDR video obtains HDR pixel values ​​and HDR format identifiers and information. HDR pixel values ​​usually need to be processed (such as image up / down sampling, enhancement, etc.). The HDR format identifier and information, as well as the processed HDR pixel values, are then sent to the encoder for re-encoding. The re-encoded HDR image or HDR video is distributed to other devices.

[0184] This solution of directly transcoding HDR images or HDR videos may cause anomalies in the transcoded HDR images or HDR videos, affecting the user experience. This is because when a single-layer encoded HDR stream is transcoded (decoded and then re-encoded), it depends on the transcoding system to correctly send all the HDR information obtained by the decoder to the encoder. If the information is lost, the transcoding system will not be able to re-encode in the correct format, resulting in an erroneous stream, causing users to have abnormal visual effects when watching the video, seriously affecting the user experience. Currently, many distribution and transcoding systems on the market have not been upgraded to support the correct transmission of HDR information from the decoder to the encoder, and mainstream mobile phone and application manufacturers have received many user complaints in this regard.

[0185] (2) Two-layer image conversion scheme.

[0186] This solution achieves dynamic range conversion of HDR images or HDR videos by converting their color gamut. The encoder transmits base layer data and enhancement layer data to the decoder, which receives and combines the base layer and enhancement layer data to produce an image with a new dynamic range supported by the display device. The color gamut of the base layer data transmitted by the encoder should match the color gamut of the display device. If the color gamut of the base layer data transmitted by the encoder does not match the color gamut of the display device, the encoder converts the color gamut of the base layer data to obtain converted base layer data that matches the color gamut of the display device and transmits this converted base layer data to the display device. For example, if the color gamut of the base layer data is P3 and the color gamut of the display device is Rec.2020, the encoder converts the color gamut of the base layer data from P3 to Rec.2020 to obtain converted base layer data. The encoder then sends the converted base layer data and enhancement layer data to the decoder. For example, the encoder can use a color gamut mapping matrix to convert the color gamut of the base layer data from P3 to Rec.2020.

[0187] This solution for adjusting the color gamut of HDR images or videos requires re-encoding and transmission adaptation if the color gamut of the display device changes, which is inflexible. Furthermore, the conversion process only adjusts the color gamut, resulting in low encoding efficiency.

[0188] Based on this, the present application provides a method for data processing. The method includes: an encoder obtains a first image, and the encoder determines the basic layer data, enhancement layer data and metadata of the first image based on the first image. The encoder encodes the basic layer data, enhancement layer data and metadata to obtain a code stream. The decoder receives and parses the code stream to obtain the basic layer data, enhancement layer data and metadata of the first image. And the decoder determines and converts the target data (i.e., the basic layer data or the enhancement layer data or the image obtained by combining the basic layer data and the enhancement layer data) according to the conversion information carried in the metadata. In this way, the decoder determines and converts the target data according to the conversion information carried in the metadata, so that the conversion of the target data matches the target data, and the decoder uses the conversion information matching the target data to convert the target data, so that the dynamic range of the converted image can not only be more adaptable to the dynamic range supported by the display device, but also improve the display effect.

[0189] The encoder, also known as the encoding side, encoding device, or encoding end, encodes the acquired image to obtain a bitstream.

[0190] The decoder, also known as the decoding side, decoding device, or decoding end, decodes the received bitstream to produce an image.

[0191] Base layer data, also referred to as a base layer image (baseline image), base layer, base data, base image, base image, base data, base layer, base layer data, or base image, can be an SDR image or an HDR image with a lower dynamic range.

[0192] Enhancement layer data, also known as gain map, enhancement layer, enhanced image, gain layer, gain layer data, gain layer image, or gain image, may include some image detail information. This information is then used to supplement the base layer data, resulting in a more HDR image with improved display quality.

[0193] For ease of description, the terms "SDR object" and "HDR object" are used in some embodiments of this application. An SDR object can be a static SDR image (also known as an SDR image, SDR photo, SDR picture, etc.), an SDR video, or other types of dynamic SDR images, or a frame within an SDR video or dynamic SDR image. An HDR object can be a static HDR image (also known as an HDR image, HDR photo, HDR picture, etc.), an HDR video, or other types of dynamic HDR images, or a frame within an HDR video or dynamic HDR image.

[0194] It is understood that for ease of description, some embodiments of the present application use HDR images as an example to introduce the technical solutions of the present application. However, it is understood that these embodiments can be applied not only to HDR images, but also to other HDR objects, such as HDR videos, dynamic HDR images, and a frame of an HDR video or dynamic HDR image.

[0195] If you need to get a better experience for HDR objects, it is an end-to-end process. Figure 5 is a schematic diagram of the end-to-end process of the HDR object provided by this application. As shown in Figure 5, the original video file (also called a master file) is obtained through material production (such as shooting video, making computer graphics (CG) video, etc.), editing, color grading and other programs. Then, based on the dynamic metadata corresponding to the original video file. After encoding the original video file and the dynamic metadata, a compressed video is obtained. The compressed video is distributed / transmitted (the distribution end is such as a streaming media server, CDN, etc., and the distribution end often needs to be transcoded, that is, the process of decoding first and then encoding) to the terminal device (for example, a computer, set-top box, mobile phone, tablet computer, etc.). The terminal device decodes the compressed video to obtain a decompressed video, and then displays it to the user through a display device (for example, a monitor, a TV, etc.).

[0196] The end-to-end process of the HDR object described in Figure 5 can be completed using the system shown in Figure 6. Figure 6 is a schematic diagram of the end-to-end system architecture of the HDR object provided by this application. As shown in Figure 6, the system includes: front-end equipment, transmission links, and terminal display equipment.

[0197] Among them, the front-end device is used to capture or produce HDR content or SDR content (for example, HDR video or image, SDR video or image).

[0198] In a possible embodiment, the front-end device can also be used to extract corresponding metadata from the HDR content. The metadata may include conversion information and dynamic metadata and static metadata corresponding to the HDR content. The front-end device can send the HDR content and metadata to the terminal display device via a transmission link. Specifically, the HDR content and metadata can be transmitted in the form of one data packet, or respectively in two data packets, which is not specifically limited in the embodiment of the present application.

[0199] In one possible embodiment, a terminal display device may be used to receive metadata and HDR content, extract conversion information contained in the corresponding metadata and information about the terminal display device based on the HDR content, obtain conversion position information and conversion scheme information, convert the HDR content, convert it into display content adapted for an HDR display device or an SDR display device in the terminal display device, and display it. It should be understood that in different embodiments, the terminal display device may include a display device with a display capability of a lower dynamic range or a higher dynamic range than the HDR content generated by the front-end device, and this application is not limited thereto.

[0200] In one possible embodiment, the front-end device and the terminal display device in this application may be independent and different physical devices. For example, the front-end device may be a video capture device or a video production device, where the video capture device may be a video camera, a camera, an image rendering machine, or the like. The terminal display device may be a device with video playback capabilities, such as virtual reality (VR) glasses, a mobile phone, a tablet, a television, or a projector.

[0201] In a possible embodiment, the transmission link between the front-end device and the terminal display device can be a wireless connection or a wired connection, wherein the wireless connection can adopt technologies such as long term evolution (LTE), fifth generation (5G) mobile communications, and future mobile communications. The wireless connection can also include wireless fidelity (WiFi), Bluetooth, near field communication (NFC), and other technologies. The wired connection can include an Ethernet connection, a local area network connection, etc. There is no specific limitation on this.

[0202] The present application may also integrate the functions of the front-end device and the terminal display device into the same physical device, for example, a terminal device such as a mobile phone or tablet with a video capture function. The present application may also integrate some functions of the front-end device and some functions of the terminal display device into the same physical device. This is not specifically limited.

[0203] Figure 7 is a schematic diagram of an end-to-end image processing system provided by this application. In Figure 7, HDR content or SDR content is exemplified by HDR video. The image processing system includes: an HDR pre-processing module, an HDR video encoding module, an HDR video decoding module, and a conversion module.

[0204] Among them, the HDR preprocessing module and the HDR video encoding module can be located in the front-end device shown in Figure 6, and the HDR video decoding module and the conversion module can be located in the terminal display device shown in Figure 6.

[0205] HDR pre-processing module: used to extract dynamic metadata (e.g., color mode, chroma format, channel information, conversion position, conversion scheme, etc.) from HDR video, and determine a conversion scheme based on the dynamic metadata and the display capabilities of the display device used to display the image processed by the image processing system described in FIG. The HDR pre-processing module obtains HDR metadata based on the dynamic metadata determined by the conversion scheme and transmits it. The HDR video can be captured or HDR video processed by a colorist; the display capability of the target display device is the brightness range that the target display device can display.

[0206] HDR video encoding module: used to encode HDR video and HDR metadata according to the video compression standard (for example, AVS or HEVC standard) (for example, embedding HDR metadata into the user-defined part of the bitstream) and output the corresponding bitstream (AVS or HEVC bitstream).

[0207] HDR video decoding module: used to decode the generated bitstream (such as AVS bitstream, AVC / H.264 bitstream, HEVC / H.265 bitstream, VVC / H.266 bitstream, etc.) according to the standard corresponding to the bitstream format, and output the decoded HDR video and HDR metadata.

[0208] Conversion module: used to determine the conversion scheme according to the conversion information in the decoded HDR metadata, and convert the decoded HDR video (i.e. HDR adaptation processing or SDR adaptation processing), and send the converted HDR adapted video to the HDR display terminal for display, or send the SDR adapted video to the SDR display terminal for display.

[0209] Exemplarily, the HDR pre-processing module may exist in a video acquisition device or a video production device.

[0210] Exemplarily, the HDR video encoding module may exist in a video acquisition device or a video production device.

[0211] Exemplarily, the HDR video decoding module may be present in a set-top box, a television display device, a mobile terminal display device, and a video conversion device for live broadcasting, online video applications, and the like.

[0212] Exemplarily, the conversion module may be present in a set-top box, a television display device, a mobile terminal display device, or a video conversion device for a live webcast or online video application. More specifically, the conversion module may be present in the form of a chip or software program in a set-top box, a television display device, or a mobile terminal display device, and may be present in the form of a software program in a video conversion device for a live webcast or online video application.

[0213] In a possible embodiment, when the conversion module and the HDR video decoding module are both present in a set-top box, the set-top box can complete the functions of receiving, decoding, and converting the video stream. The set-top box sends the decoded video data to a display device through a high-definition multimedia interface (HDMI) for display, so that the user can enjoy the video content.

[0214] The end-to-end process described in Figures 5 to 7 above can be completed using a streaming media system. Figure 8 is a schematic diagram of the workflow of a streaming media system provided by the present application. As shown in Figure 8, the streaming media system includes: a content creation module, a video encoding module, and a video stream transmission module. The content creation module generates the required content data, such as video or audio. The video encoding module encodes the content data generated by the content creation module through an encoder. The video stream transmission module transmits the encoded video in the form of a code stream.

[0215] In some possible scenarios, the streaming media system further includes a video stream transcoding module. The video stream transcoding module can convert the format of the video stream into a code stream format of a commonly used transmission protocol for OTT (over-the-top) devices, for example, the protocol includes but is not limited to Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP), or any combination thereof.

[0216] In some possible situations, the streaming media system further includes a video stream storage module, which stores the original format of the video stream and / or the converted multiple code stream formats for easy use.

[0217] In some possible scenarios, the streaming media system further includes a video stream encapsulation module. The video stream encapsulation module encapsulates the video stream to generate an encapsulated video stream, which can be referred to as a video streaming media package. Exemplarily, the video streaming media package can be generated based on a transcoded video stream or a stored video stream.

[0218] In some possible scenarios, the streaming media system also includes a content distribution network (CDN). The CDN is used to distribute video streaming packages to multiple OTT devices, such as mobile phones, computers, tablets, home projectors, etc.

[0219] It should be noted that video encoding, video stream transmission, video stream transcoding, video stream storage, video streaming package generation, and CDN can all be implemented using the cloud platform. Figure 9 is a schematic diagram of a streaming media system architecture provided by this application. As shown in Figure 9, the streaming media system includes: a client device (or simply a client), a CDN, and a cloud platform.

[0220] The user on the client sends a play or playback request to the cloud platform. In some possible examples, the content of the request may be the title of the movie or TV program to be played.

[0221] The cloud platform makes a decision and responds to the client, sending the address of the content requested by the user on the CDN to the client. In some possible examples, the content sent to the client can be a URL (uniform resource locator) link. Specifically, the playback application service in the cloud platform checks user authorization and permissions, and then considers the characteristics of each client and the current network conditions to determine which specific files are needed to process the playback request. It should be noted that the content delivery network (CDN) regularly reports its operating status, learned routes, and available content (files) to the cache control service in the cloud platform.

[0222] The client then requests the CDN to play the content based on the address, and the CDN provides the content to the client, ultimately completing the user's request.

[0223] The above text describes the process of front-end device acquisition, transmission link or CDN distribution / transmission, and display device display of HDR objects or SDR objects in conjunction with the accompanying drawings. The following text describes the codec system used in this application in conjunction with the accompanying drawings. Figure 10 is a schematic block diagram of a codec system used in this application, for example, the video codec system 1000 (or simply referred to as the codec system 1000) of this application can be used. The video encoder 1015 (or simply referred to as encoder 1015) and the video decoder 1022 (or simply referred to as decoder 1022) in the video codec system 1000 represent devices that can be used to perform various technologies according to the various examples described in this application.

[0224] As shown in FIG10 , a coding and decoding system 1000 includes a source device 1010 , which is configured to provide coded data such as coded images to a target device 1020 configured to decode the coded data.

[0225] The source device 1010 includes an encoder 1015 , and optionally, may further include an image source 1011 , a preprocessor 1013 (or a preprocessing unit, an image preprocessor, etc.), and a communication interface 1017 or a communication unit 1017 .

[0226] The image source 1011 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage for storing any of the above images.

[0227] In order to distinguish the processing performed by the pre-processor 1013 or the pre-processing unit 1013 , the image or image data 1012 may also be referred to as a raw image or raw image data 1012 .

[0228] The preprocessor 1013 is configured to receive (raw) image data 1012 and preprocess the image data 1012 to obtain a preprocessed image 1014 or preprocessed image data 1014. For example, the preprocessing performed by the preprocessor 1013 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessing unit 1013 may be an optional component.

[0229] The encoder 1015 is configured to receive the pre-processed image data 1014 and provide encoded image data 1016 .

[0230] The communication interface 1017 in the source device 1010 can be used to receive the encoded image data 1016 and send the encoded image data 1016 (or any other processed version) to another device such as the target device 1020 or any other device via the communication channel 1030 for storage or direct reconstruction.

[0231] The target device 1020 includes a decoder 1022 (eg, a video decoder 1022 (or simply decoder 1022 )), and optionally, may include a communication interface 1021 or communication unit 1021 , a post-processor 1024 (or post-processing unit 1024 ), and a display device 1026 .

[0232] The communication interface 1021 in the target device 1020 is used to receive the encoded image data 1016 (or any other processed version) directly from the source device 1010 or from any other source device such as a storage device, for example, the storage device is a coded image data storage device, and provide the encoded image data 1016 to the decoder 1022.

[0233] The communication interface 1017 and the communication interface 1021 can be used to send or receive the encoded image data 1016 or the encoded data through a direct communication link between the source device 1010 and the target device 1020, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0234] For example, the communication interface 1017 may be configured to encapsulate the encoded image data 1016 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0235] The communication interface 1021 corresponds to the communication interface 1017 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 1016 .

[0236] Both the communication interface 1017 and the communication interface 1021 can be configured as a unidirectional communication interface as indicated by the arrow pointing from the source device 1010 to the corresponding communication channel 1030 of the target device 1020 in Figure 10, or a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0237] The decoder 1022 is configured to receive the encoded image data 1016 and provide decoded image data 1023 or a decoded image 1023 .

[0238] The post-processor 1024 in the target device 1020 is configured to post-process the decoded image data 1023 (also referred to as reconstructed image data), such as the decoded image 1023, to obtain post-processed image data 1025, such as the post-processed image 1025. The post-processing performed by the post-processing unit 1024 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 1023 for display on a display device 1026, etc.

[0239] The display device 1026 in the target device 1020 is configured to receive the post-processed image data 1025 and display the image to a user or viewer. The display device 1026 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0240] Although FIG10 illustrates the source device 1010 and the target device 1020 as separate devices, device embodiments may also include both the source device 1010 and the target device 1020 or the functions of both the source device 1010 and the target device 1020, that is, include both the source device 1010 or the corresponding functions and the target device 1020 or the corresponding functions. In these embodiments, the source device 1010 or the corresponding functions and the target device 1020 or the corresponding functions may be implemented using the same hardware and / or software or through separate hardware and / or software, or any combination thereof.

[0241] According to the description, the existence and (accurate) division of different units or functions in the source device 1010 and / or the target device 1020 shown in FIG. 10 may vary depending on actual devices and applications, which is obvious to technicians.

[0242] The following describes the content providing system used by the encoding and decoding system described in Figure 10 in conjunction with Figure 11. Figure 11 is a block diagram of a content providing system provided by this application. The content providing system 1100 includes a capture device 1110, a terminal device 1120, and an (optional) display 1130. The capture device 1110 communicates with the terminal device 1120 via a communication link 1140. The communication link may include the communication channel 1030 described in Figure 10. The communication link 1140 includes, but is not limited to, WiFi, Ethernet, wired, wireless (3G / 4G / 5G), USB, or any combination thereof.

[0243] Capture device 1110 can implement the functions of the front-end device in Figure 6. Capture device 1110 generates data and can encode the data using the encoding methods described in the above embodiments. Alternatively, capture device 1110 can distribute the data to a streaming media server (not shown), which encodes the data and transmits the encoded data to terminal device 1120. Capture device 1110 includes, but is not limited to, a camera, a smartphone or tablet computer, a computer or laptop computer, a video conferencing system, a personal digital assistant (PDA), an in-vehicle device, or any combination thereof. For example, capture device 1110 can include the aforementioned source device 1010. When the data includes video, encoder 1015 in capture device 1110 can actually perform video encoding. When the data includes audio (i.e., sound), encoder 1015 in capture device 1110 can actually perform audio encoding. In some practical scenarios, capture device 1110 distributes the encoded video data and the encoded audio data by multiplexing them together. In other practical scenarios, such as in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. The capture device 1110 distributes the encoded audio data and the encoded video data to the terminal device 1120 respectively.

[0244] The terminal device 1120 in the content delivery system 1100 receives and regenerates the encoded data. The terminal device 1120 can be a device capable of receiving and recovering data, such as a smartphone or tablet 1121, a computer or laptop 1122, a network video recorder (NVR) / digital video recorder (DVR) 1123, a television 1124, a set-top box (STB) 1125, a video conferencing system 1126, a video surveillance system 1127, a PDA 1128, an in-vehicle device 1129, or any combination thereof, or any such device capable of decoding the encoded data. For example, the terminal device 1120 can include the target device 1020 described above. When the encoded data includes video, the decoder 1022 in the terminal device prioritizes video decoding. When the encoded data includes audio, the decoder 1022 in the terminal device prioritizes audio decoding. The terminal device 1120 can be a video playback application, a streaming media playback application, a streaming media playback platform, or a live broadcast platform running on the terminal device.

[0245] For end devices with displays, such as smartphones or tablets 1121, computers or laptops 1122, NVRs / DVRs 1123, televisions 1124, PDAs 1128, or in-vehicle devices 1129, the end device can send the decoded data to its display. For end devices without displays, such as STBs 1125, video conferencing systems 1126, or video surveillance systems 1127, the device can be connected to an external display 1130 to receive and display the decoded data.

[0246] When each device in this system performs encoding or decoding, the image encoding device or image decoding device shown in the above embodiment can be used. Figure 12 is a schematic diagram of the example structure of the terminal device 1120 in Figure 11. Figure 12 is a schematic diagram of the workflow of a terminal device provided by this application. As shown in Figure 12, after the terminal device 1120 receives a bit stream from the capture device 1110, the protocol processing unit 1210 analyzes the transmission protocol of the bit stream. The transmission protocol includes but is not limited to: RTSP, HTTP, HLS, MPEG-DASH, RTP, RTMP or any combination thereof.

[0247] After processing the bitstream, protocol processing unit 1210 generates a stream file. This stream file is output to demultiplexing unit 1220. Demultiplexing unit 1220 can separate the multiplexed data into coded audio data and coded video data. As mentioned above, in other practical scenarios, such as video conferencing systems, coded audio data and coded video data are not multiplexed. In this case, the coded data is not transmitted to video decoder 1230 and audio decoder 1240 through demultiplexing unit 1220.

[0248] The demultiplexing unit 1220 processes the stream file to generate a video elementary stream (ES), an audio ES, and optional subtitles. The video decoder 1230, including the decoder 1022 as described in the above embodiment, decodes the video ES using the decoding method shown in the above embodiment to generate a video frame, and sends the video frame to the synchronization unit 1260. The audio decoder 1240 decodes the audio ES to generate an audio frame, and sends the audio frame to the synchronization unit 1260. Alternatively, the video frame can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 1260. Similarly, the audio frame can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 1260.

[0249] Synchronization unit 1260 synchronizes video frames and audio frames and provides the video / audio to video / audio display 1270. For example, synchronization unit 1260 synchronizes the presentation of video and audio information. The information may be encoded in the syntax using timestamps associated with the representation of the encoded audio and visual data and timestamps associated with the transmission of the data stream.

[0250] If subtitles are included in the bitstream, the subtitle decoder 1250 decodes the subtitles, synchronizes them with the video frames and audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 1280 .

[0251] The present application is not limited to the above-mentioned system. The image encoding device or image decoding device in the above-mentioned embodiments can be used in other systems such as automobiles.

[0252] The above description, combined with the accompanying drawings, describes the encoding and decoding system provided by the present application. The following description, combined with the accompanying drawings, describes the workflow diagram for image processing provided by the present application. The image processing process can be performed by an electronic device, which includes a decoding module and a processing module. In some possible examples, the electronic device may also include a display module for displaying the obtained target image.

[0253] The decoding module can include any decoder (HEVC, JPEG, ProRes, HEIF). The decoding module uses the decoder to decode the received code stream to obtain base layer data, enhancement layer data and metadata. The base layer data and enhancement layer data obtained in this process include image data in any color space form such as RGB or YUV. It should be noted that this application does not limit the code stream format. In terms of color space, it can be YUV or RGB. In terms of data bit width, the code stream can be 8 bits, 10 bits, or 12 bits, etc. In some embodiments, the decoding module can obtain source data from the SEI of HEVC or VVC, user-defined NAL units or reserved packet units, or obtain the metadata from JFIF-encapsulated APP extension information or MP4-encapsulated data segments, etc.; it can also obtain the metadata from other file locations, such as the location after the EOI (end of image) of the complete JPEG file.

[0254] It should be noted that metadata mainly includes the source data format or image characteristics and conversion information, as well as one or more metadata information units. Metadata information units include coordinate information, image characteristics and conversion information.

[0255] In some embodiments, the decoding module can obtain enhancement layer data from the SEI of HEVC or VVC, user-defined NAL units or reserved packet units, or from JFIF-encapsulated APP extension information or MP4-encapsulated data segments; it can also obtain enhancement layer data from other file locations indicated in the metadata, such as the position after the EOI of the complete JPEG file.

[0256] The enhancement layer data may be: one pixel value corresponds to one enhancement layer data, or multiple pixel values ​​correspond to one enhancement layer data, wherein the enhancement layer data may contain multiple values, and the number of values ​​contained in the enhancement layer data corresponding to each pixel is the same.

[0257] The electronic device can use a decoding module or a processing module to perform different processing on the base layer data and the enhancement layer data to obtain a target image, which is described below in different situations.

[0258] Scenario 1. FIG13A is a schematic diagram of the first image processing process provided by this application. As shown in FIG13A , the decoding module uses a decoder to obtain base layer data, enhancement layer data, and metadata from the bitstream. The processing module uses the metadata to process the base layer data to obtain processed base layer data, and then combines the processed base layer data and enhancement layer data to obtain a target image. The processing module transmits the target image to the display module for display.

[0259] Scenario 2, FIG13B is a schematic diagram of the second image processing process provided by this application. As shown in FIG13B , the decoding module uses a decoder to obtain base layer data, enhancement layer data, and metadata from the bitstream. The decoding module synthesizes the base layer data and enhancement layer data to obtain an HDR image (or HDR image 1), and transmits HDR image 1 to the processing module. The processing module performs processing 1 on HDR image 1 to obtain HDR image 2. The processing module performs processing 2 on HDR image 2 using the metadata to obtain HDR image 3. The processing module transmits HDR image 3 to the display module for display.

[0260] The dynamic range of the base layer data is smaller than the dynamic range of the HDR image (ie, HDR image 1) determined based on the base layer data and the enhancement layer data.

[0261] For example, in some embodiments, the base layer data may be an SDR image, and the HDR image 1 may be an HDR image. The HDR image and the SDR image may have at least one of a different photoelectric conversion function, a different color gamut, and a different dynamic range.

[0262] For another example, in some other embodiments, the base layer data may be an HDR image, but the dynamic range of the HDR image is smaller than the dynamic range of the HDR image 1 .

[0263] For example, assume that the dynamic range of the base layer data is DRx1 to DRx2, and the dynamic range of HDR Image 1 is DRy1 to DRy2. In some embodiments, DRy1 is less than DRx1, and DRy2 is greater than DRx2. In other embodiments, DRy1 is less than DRx1, and DRy2 is equal to DRx2. In other embodiments, DRy1 is equal to DRx1, and DRy2 is greater than DRx2. In other embodiments, DRy1 is equal to DRx1, and DRy2 is greater than DRx2. In other words, the base layer data can be considered to have an image with a smaller dynamic range than HDR Image 1.

[0264] In some embodiments, the resolution of the base layer data may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0265] In some embodiments, the resolution of the enhancement layer data may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0266] In some embodiments, the resolution of the HDR image 1 may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0267] In some embodiments, the resolution of the HDR image 2 may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0268] In some embodiments, the resolution of the HDR image 3 may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0269] In some embodiments, the resolution of the base layer data may be the same as the resolution of the enhancement layer data, for example, the resolution of both the base layer data and the enhancement layer data may be 4K.

[0270] In other embodiments, the resolution of the base layer data may be different from the resolution of the enhancement layer data. For example, in some embodiments, the resolution of the base layer data may be greater than the resolution of the enhancement layer data. For example, the resolution of the base layer data may be 4K, and the resolution of the enhancement layer data may be 2K or 1080P. For another example, in other embodiments, the resolution of the base layer data may be smaller than the resolution of the enhancement layer data. For example, the resolution of the base layer data may be 1080P or 2K, and the resolution of the enhancement layer data may be 4K.

[0271] Scenario 3. FIG13C is a schematic diagram of the third image processing process provided by this application. As shown in FIG13C , the decoding module uses a decoder to obtain base layer data, enhancement layer data, and metadata from the bitstream, and the processing module performs processing 1 on the base layer data using the metadata to obtain processed base layer data. The processing module also performs processing 2 on the enhancement layer data using the metadata to obtain processed enhancement layer data. The processing module synthesizes the processed base layer data and the processed enhancement layer data to obtain an HDR image (i.e., the target image). The processing module transmits the HDR image to the display module for display.

[0272] The above text, in combination with the accompanying drawings, describes the process of performing image processing by the electronic device provided in this application. In some possible situations, modules deployed on different electronic devices can also be used to implement the above image processing process. Three possible examples are given below.

[0273] In Example 1, Figure 13D is a schematic diagram of the fourth image processing process provided by this application. As shown in Figure 13D, the decoding module and the processing module cooperate to implement the image processing process to obtain a target image, and the display module displays the target image. The decoding module decodes the received bitstream to obtain a base image, a gain map, and metadata. The processing module uses the metadata to process the base image to obtain a processed base image. The processing module then obtains the target image based on the processed base image and the gain map.

[0274] Example 2, Figure 13E is a flow chart of the fifth image processing provided by this application. As shown in Figure 13E, the decoding module decodes the received code stream to obtain the base image, enhancement layer and metadata, and synthesizes the enhancement layer and the base layer to obtain an HDR image synthesis. The image processing module processes the HDR image synthesis 2 to obtain a processed image 1, and transmits the processed image 1 to the display module. The image processing module processes the base layer 1 to obtain a processed image 2, and transmits the processed image 2 to the display module. The display module receives the processed image 1 and the processed image 2, obtains the target image based on the processed image 1 and the processed image 2, and displays the target image.

[0275] Example 3, FIG13F is a schematic diagram of the sixth image processing process provided by this application. As shown in FIG13F , base layer data, metadata, and enhancement layer data are obtained by decoding the bitstream, or directly obtained from the bitstream. Process 1 is performed on the base layer data to obtain processed image 1, and process 2 is performed on the enhancement layer data to obtain processed image 2. Processed image 1 and processed image 2 are then synthesized, and the synthesized image is displayed.

[0276] The above describes the general process of image processing in conjunction with the accompanying drawings. The following describes a data processing method provided by the present application in conjunction with the accompanying drawings. Figure 14 is a flow chart of a data processing method provided by the present application. As shown in Figure 14, the method includes the following S1410 to S1460.

[0277] S1410: The encoder determines, according to the acquired first image, base layer data of the first image, enhancement layer data of the first image, and metadata of the first image.

[0278] The metadata carries conversion information.

[0279] In a possible scenario, the conversion information includes at least one of color mode information, chroma format information, color gamut information, and color compensation information.

[0280] The color mode information includes: color mode information of the basic layer data of the first image, color mode information of the enhanced layer data of the first image, and at least one of the color mode information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0281] The chroma format information includes: chroma format information of the basic layer data of the first image, chroma format information of the enhanced layer data of the first image, and at least one of the chroma format information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0282] The color gamut information includes: color gamut information of the basic layer data of the first image, color gamut information of the enhanced layer data of the first image, and at least one of the color gamut information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0283] The color compensation information includes: color compensation information of the basic layer data of the first image, color compensation information of the enhanced layer data of the first image, and at least one of the color compensation information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0284] In one possible scenario, the conversion information further includes conversion location information and conversion scheme information. The conversion location information indicates the location where the conversion occurs. The conversion scheme information indicates the conversion scheme used when converting the target data. The conversion occurs before or after obtaining the second image. The second image indicates an image obtained based on the base layer data and the enhancement layer data.

[0285] The content of the conversion information varies depending on where the conversion occurs, and is described below in different situations.

[0286] In case A, the conversion occurs before obtaining the second image.

[0287] In this case, the conversion information also includes conversion object information. The conversion object information is used to indicate the conversion object for which the conversion scheme is to be executed. The conversion object includes: at least one of base layer data and enhancement layer data, for example, the conversion object includes base layer data, or the conversion object includes enhancement layer data, or the conversion object includes base layer data and enhancement layer data.

[0288] The content of the conversion plan varies depending on the conversion object, and is described below in different situations.

[0289] In case Aa, the conversion object includes basic layer data.

[0290] In this case, the conversion scheme information includes: a base layer data conversion scheme. The base layer data conversion scheme is used to indicate whether to perform at least one of color gamut conversion, chroma format conversion, color compensation, and color mode conversion on the base layer data. Performing color gamut conversion on the base layer data may refer to performing color gamut conversion on the base layer data using the color gamut information carried in the metadata corresponding to the transformation and processing of the base layer data and the enhancement layer data, or using preset color gamut information. Performing color mode conversion on the base layer data may refer to performing color mode conversion on the base layer data using the color mode information carried in the metadata corresponding to the transformation and processing of the base layer data and the enhancement layer data, or using preset color mode information. Performing chroma format conversion on the base layer data may refer to performing color mode conversion on the base layer data using the chroma format information carried in the metadata corresponding to the transformation and processing of the base layer data and the enhancement layer data, or using preset chroma format information. Color compensation of base layer data may refer to color compensation of base layer data using color compensation information in transformation and processing corresponding to base layer data and enhancement layer data carried in metadata, or color compensation of base layer data using preset color compensation information.

[0291] The base layer data conversion scheme is further used to indicate conversion parameters used when converting the base layer data. The conversion parameters include at least one of a compensation matrix, a conversion matrix, a conversion function, and a compensation value.

[0292] Case Ab: the conversion object includes enhancement layer data.

[0293] In this case, the conversion scheme information includes: an enhancement layer data conversion scheme, wherein the enhancement layer data conversion scheme is used to indicate that at least one of chroma format conversion, color mode conversion, and channel conversion is performed on the enhancement layer data.

[0294] For details about chroma format conversion and color mode conversion, please refer to the description in the above scenario Aa, which will not be repeated here. Channel conversion is used to restore the channels of the enhancement layer data to the original channels.

[0295] The enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the enhancement layer data. The conversion parameters include: at least one of a conversion matrix, a conversion function, and a channel conversion function.

[0296] When the conversion target is base layer data, the electronic device may convert the base layer data using the base layer data conversion scheme described in scenario Aa. When the conversion target is enhancement layer data, the electronic device may convert the enhancement layer data using the enhancement layer data conversion scheme described in scenario Ab. Furthermore, when the conversion targets are base layer data and enhancement layer data, the electronic device may convert the base layer data using the base layer data conversion scheme described in scenario Aa, and convert the enhancement layer data using the enhancement layer data conversion scheme described in scenario Ab.

[0297] In case B, the conversion occurs after the second image is obtained.

[0298] In this case, the conversion scheme information includes at least one of: color gamut conversion, chroma format conversion, color compensation, and color mode conversion. For details about color gamut conversion, chroma format conversion, color compensation, and color mode conversion, please refer to the above description and will not be repeated here.

[0299] In one possible scenario, the conversion scheme information further includes conversion parameters. The conversion parameters include at least one of a compensation matrix, a conversion matrix, a conversion function, and a compensation value. For a description of the compensation matrix, the conversion matrix, the conversion function, and the compensation value, please refer to conventional techniques and are not further described here.

[0300] In one possible scenario, when the target data includes base layer data and enhancement layer data, the conversion information may further include channel merging information. The channel merging information is used to indicate a scheme for combining channels of the base layer data and channels of the enhancement layer data. The scheme for combining the base layer data and the enhancement layer data varies depending on the color mode of the base layer data and the color mode of the enhancement layer data. In this case, the channel merging information may be used to indicate a scheme for combining channels corresponding to the color mode of the base layer data and channels corresponding to the color mode of the enhancement layer data. Depending on the color mode of the base layer data and the color mode of the enhancement layer data, the schemes for combining the base layer data and the enhancement layer data are different, which are described below.

[0301] When the color modes of the basic layer data and the enhanced layer data are both RGB modes, the channel merging information is used to indicate: the R channel of the basic layer data and the R channel of the enhanced layer data are merged, the G channel of the basic layer data and the G channel of the enhanced layer data are merged, and the B channel of the basic layer data and the B channel of the enhanced layer data are merged.

[0302] When the color mode of the basic layer data is RGB mode and the color mode of the enhanced layer data is grayscale mode, the channel merging information is used to indicate: the R channel of the basic layer data is merged with the grayscale value of the enhanced layer data, the G channel of the basic layer data is merged with the grayscale value of the enhanced layer data, and the B channel of the basic layer data is merged with the grayscale value of the enhanced layer data.

[0303] When the color mode of the basic layer data is the grayscale mode and the color mode of the enhanced layer data is the RGB mode, the channel merging information is used to indicate: the R channel of the enhanced layer data is merged with the grayscale value of the basic layer data, the G channel of the enhanced layer data is merged with the grayscale value of the basic layer data, and the B channel of the enhanced layer data is merged with the grayscale value of the basic layer data.

[0304] In the case that the color modes of the base layer data and the enhancement layer data are both grayscale modes, the channel merging information is used to indicate that the grayscale value of the base layer data and the grayscale value of the enhancement layer data are merged.

[0305] When the color mode of the basic layer data and the color mode of the enhanced layer data are both any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0306] When the color modes of the basic layer data and the enhanced layer data are any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, the channel merging information is used to indicate that the brightness channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0307] When the color mode of the basic layer data is grayscale mode and the color mode of the enhanced layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, the channel merging information is used to indicate: the channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0308] When the color mode of the basic layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, and the color mode of the enhanced layer data is the grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the channel of the enhanced layer data are merged.

[0309] There are multiple ways to merge channels, which may be preset or carried by metadata. Several possible ways to merge channels are given below.

[0310] Method 1: The channels of the base layer data and the channels of the enhancement layer data are directly multiplied.

[0311] In this case, the decoder can use formula (6) to merge the channels of base layer data and enhancement layer data. newPictue[i]=Base[i]*Enhance[i] Formula (6)

[0312] Where newPictue[i] refers to the merged image, Base[i] refers to the base layer data, and Enhance[i] refers to the enhancement layer data.

[0313] Method 2: The channels of the base layer data and the channels of the enhancement layer data are directly added.

[0314] In this case, the decoder can use formula (7) to merge the channels of base layer data and enhancement layer data. newPictue[i]=Base[i]+Enhance[i] Formula (7)

[0315] Where newPictue[i] refers to the merged image, Base[i] refers to the base layer data, and Enhance[i] refers to the enhancement layer data.

[0316] Method 3: directly multiply the channels of the base layer data and the channels of the processed enhancement layer data.

[0317] In this case, the decoder can use formula (8) to merge the channels of base layer data and enhancement layer data. newPictue[i] = Base[i] * f(Enhance[i]) Formula (8)

[0318] Where newPictue[i] is the merged image. Base[i] is the base layer data. Enhance[i] is the enhancement layer data. f() is the domain conversion function.

[0319] Method 4: The channels of the base layer data and the channels of the processed enhancement layer data are directly added.

[0320] In this case, the decoder can use formula (9) to merge the channels of base layer data and enhancement layer data. newPictue[i]=Base[i]+f(Enhance[i]) Formula (9)

[0321] Where newPictue[i] is the merged image. Base[i] is the base layer data. Enhance[i] is the enhancement layer data. f() is the domain conversion function.

[0322] Method 5: directly multiply the channels of the base layer data and the channels of the enhancement layer data after the first processing, and perform the second processing on the obtained results.

[0323] In this case, the decoder can use formula (10) to merge the channels of the base layer data and the channels of the enhancement layer data. newPictue[i] = g(Base[i]*f(Enhance[i])) Formula (10)

[0324] Where newPictue[i] is the merged image. Base[i] is the base layer data. Enhance[i] is the enhancement layer data. f() is the domain conversion function, and g() is the mapping function.

[0325] Method 6: directly add the channels of the base layer data and the channels of the enhancement layer data after the first processing, and perform the second processing on the obtained results.

[0326] In this case, the decoder can use formula (11) to merge the channels of base layer data and enhancement layer data. newPictue[i] = g(Base[i] + f(Enhance[i])) Formula (11)

[0327] Where newPictue[i] is the merged image. Base[i] is the base layer data. Enhance[i] is the enhancement layer data. f() is the domain conversion function, and g() is the mapping function.

[0328] According to the needs of actual applications, the numerical domain conversion function f() and the mapping function g() in Methods 1 to 6 can be any functions, and this application does not limit them, such as f(x)=x, g(x)=x.

[0329] The following describes a method for merging channels of base layer data and enhancement layer data, using the multiplication method described in Method 1 and the addition method described in Method 2 as examples. Channel merging may also be performed in other merging methods, such as those described in Methods 3 through 6 above. Depending on actual application needs, channels may be merged in other methods besides Methods 1 through 6, and this application does not limit this.

[0330] The channel merging methods are different depending on the number of channels of the base layer data and the number of channels of the enhancement layer data, which are described below respectively.

[0331] Case 1: base layer data is a single channel, and enhancement layer data is a single channel.

[0332] In this case, the channel merging information indicates whether the single channel of the base layer data and the single channel of the enhancement layer data are merged using the channel merging method described above. For example, the base layer data channel and the enhancement layer data channel are directly merged. Using the merging methods described in Examples 1 and 2 as an example, the channel merging information may indicate the content shown in Table 1.

[0333] Table 1

[0334] When image 1 is base layer data, image 2 may be enhancement layer data, or when image 1 is enhancement layer data, image 2 may be base layer data.

[0335] Case 2: The base layer data is multi-channel and the enhancement layer data is single-channel.

[0336] In this case, the channel combining scheme indicated by the channel merging information is different according to the color mode of the basic layer data.

[0337] (1) A color mode in which the brightness and chrominance of the base layer data are not separated.

[0338] In this case, the channel merging information indicates: merging the channels of the enhancement layer data with the channels of the base layer data. Taking the base layer data in RGB mode as an example, the content indicated by the channel merging information may be the content in Table 2.

[0339] Table 2

[0340] When image 1 is base layer data, image 2 may be enhancement layer data.

[0341] (2) The basic layer data is a color model with brightness and chrominance separated.

[0342] In this case, the channel combining scheme indicated by the channel merging information is different depending on whether the channel of the enhancement layer data is used to enhance the channels of the basic layer data other than the brightness channel (such as the chroma channel, the hue channel, etc.).

[0343] ① Use the channels of the enhancement layer data to process the channels of the basic layer data except the brightness channel.

[0344] In this case, the channel merging information indicates that the channel of the enhancement layer data is merged into the luminance channel of the base layer data and at least one other channel except the luminance channel.

[0345] For example, the channel merging information indicates: merge the channel of the enhancement layer data into each channel of the basic layer data. Taking the basic layer data as YUV mode and the channel of the enhancement layer data as C as an example, the channel merging information indicates that the channel C of the enhancement layer data is merged into the Y channel, U channel and V channel of the basic layer data, which can be represented by the content corresponding to the row number 1 in Table 3.

[0346] For another example, the channel merging information indicates that the channel of the enhancement layer data is merged with the luma and chroma channels of the base layer data. For example, if the base layer data is in YUV mode and the channel of the enhancement layer data is C, the channel merging information indicates that the channel C of the enhancement layer data is merged with the Y and U channels of the base layer data. This can be represented by the content corresponding to row number 2 in Table 3.

[0347] ② Use the channel of the enhancement layer data to process the brightness channel of the basic layer data.

[0348] The channel merging information indicates: Merge the channels of the enhancement layer data into the luma channel of the base layer data. For example, if the base layer data is in YUV mode and the channel of the enhancement layer data is C, the content of the channel merging information indication may be the content corresponding to row number 3 in Table 3.

[0349] Table 3

[0350] When image 1 is base layer data, image 2 may be enhancement layer data.

[0351] Case 3: base layer data is multi-channel, and enhancement layer data is multi-channel.

[0352] In this case, the base layer data and the enhancement layer data can both use separate luma and chroma color modes, or both use unseparated luma and chroma color modes, or one uses separate luma and chroma color modes while the other does not. In each case, the channel merging information differs, as described below.

[0353] (1) Both the base layer data and the enhancement layer data are not in a mode where luminance and chrominance are not separated.

[0354] The base layer data and the enhancement layer data can be in the same color mode or in two different color modes, as illustrated below with examples.

[0355] ① The basic layer data and the enhanced layer data use the same color mode.

[0356] In this case, the channel merging information indicates that at least one channel of the enhancement layer data is to be merged with the corresponding channel of the base layer data. For example, if the channel merging information indicates that each channel of the enhancement layer data is to be merged with the corresponding channel of the base layer data, and both the base layer data and the enhancement layer data are in RGB mode, RGBA mode, or CMYK mode, the contents indicated by the channel merging information may be those shown in Table 4.

[0357] Table 4

[0358] When image 1 is base layer data, image 2 may be enhancement layer data.

[0359] The above takes the example of merging each channel of the base layer data and the corresponding channel of the enhancement layer data. In some possible situations, the channel merging information can also indicate that some channels of the enhancement layer data are merged with the corresponding channels of the base layer data.

[0360] For example, the basic layer data and the enhanced layer data are in RGBA mode, and the channel merging information indicates: the R channel of the basic layer data and the R channel of the enhanced layer data are merged, the G channel of the basic layer data and the G channel of the enhanced layer data are merged, and the B channel of the basic layer data and the R channel of the enhanced layer data are merged.

[0361] ② There are two different color modes for basic layer data and enhanced layer data.

[0362] In this case, the channel merging information indicates that each channel of the enhancement layer data is merged with the corresponding channel of the base layer data. For example, the base layer data is in RGB mode and the enhancement layer data is in RGBA mode. In this case, the channel merging information may indicate the content shown in Table 5.

[0363] Table 5

[0364] When image 1 is base layer data, image 2 may be enhancement layer data, or when image 1 is enhancement layer data, image 2 may be base layer data.

[0365] (2) The base layer data and the enhancement layer data are both in color modes with brightness and chrominance separated.

[0366] The channel merging information indicates at least one of merging the luminance channel of the enhancement layer data and the luminance channel of the base layer data, and merging the chrominance channel of the enhancement layer data and the chrominance channel of the base layer data.

[0367] ① The basic layer data and the enhanced layer data use the same color mode.

[0368] In this case, the channel merging information may indicate that the luma channel of the enhancement layer data is to be merged with the luma channel of the base layer data. For example, if the color modes of the base layer data and the enhancement layer data are both YUV, Lab, or HSV, and the channel merging information indicates that only the luma channel of the enhancement layer data is to be merged with the luma channel of the base layer data, the channel merging information may be represented by the content shown in Table 6.

[0369] Table 6

[0370] The above takes the example of channel merging information indicating the merging of the luminance channel of the enhancement layer data and the luminance channel of the base layer data. In some possible situations, the channel merging information can also indicate: the luminance channel of the enhancement layer data and at least one of the other channels are merged with the corresponding channel of the base layer data.

[0371] For example, the color modes of the basic layer data and the enhanced layer data are both YUV modes, and the channel merging information can indicate that the Y channel of the enhanced layer data is merged into the Y channel of the basic layer data, the U channel of the enhanced layer data is merged into the U channel of the basic layer data, and the V channel of the enhanced layer data is merged into the V channel of the basic layer data.

[0372] For example, the color modes of the basic layer data and the enhanced layer data are both Lab modes, and the channel merging information can indicate that the L channel of the enhanced layer data is merged into the L channel of the basic layer data, the a channel of the enhanced layer data is merged into the a channel of the basic layer data, and the b channel of the enhanced layer data is merged into the b channel of the basic layer data.

[0373] For another example, the color modes of the basic layer data and the enhanced layer data are both HSV modes, and the channel merging information can indicate that the H channel of the enhanced layer data is merged into the H channel of the basic layer data, the S channel of the enhanced layer data is merged into the S channel of the basic layer data, and the V channel of the enhanced layer data is merged into the V channel of the basic layer data.

[0374] ② There are two color modes for basic layer data and enhanced layer data.

[0375] For example, the basic layer data is in YUV mode, and the enhanced layer data is in Lab mode. For another example, the basic layer data is in HSV mode, and the enhanced layer data is in Lab mode. For another example, the basic layer data is in YUV mode, and the enhanced layer data is in HSV mode. In this case, it is necessary to convert the color modes of the basic layer data and the enhanced layer data, and merge the channels of the basic layer data and the enhanced layer data after the color mode conversion. For example, the color mode of the basic layer data and the color mode of the enhanced layer data are both converted into RGB mode to obtain the converted basic layer data in RGB mode and the enhanced layer data in RGB mode. In this case, the channel merging information corresponding to sequence number 1 in Table 4 above can be used to merge the converted basic layer data and the converted enhanced layer data. (3) One of the basic layer data and the enhanced layer data is in a color mode with luminance and chrominance separated, and the other is in a color mode that is not a luminance and chrominance separated.

[0376] In this case, the channel merging information indicates that the color mode of at least one of the base layer data and the enhancement layer data is converted using the conversion information, and that the channels are merged using the methods described in (1) and (2) above for the converted data. For the same contents of the channel merging information for the converted base layer data and the enhancement layer data, please refer to the relevant descriptions in (1) and (2) above, which will not be repeated here.

[0377] S1420: The encoder encodes the base layer data, the enhancement layer data, and the metadata to obtain a code stream.

[0378] Metadata can be used to define the image's aspect ratio. Image resolution doesn't necessarily represent aspect ratio. If the pixel distribution on the display isn't spatially uniform—that is, the horizontal and vertical distances between pixels differ—this can lead to phase errors between different interpolation algorithms. For example, the actual distances between adjacent horizontal and vertical pixels that differ by 1 / 2 of their phase can differ. Metadata is used to determine the range of actual values ​​stored in the image, or the upper and lower limits of the dynamic range. For example, when reading an HDR image during image input and output, it may have a 10-bit bit width, but the image falls within the limited range. This limited range is specified in documents such as BT.2020 as 64-940 for Luma and 64-960 for Chroma (for 10-bit). In practice, the upper and lower ranges of this bit width should be indicated by metadata and should be the larger of the bit widths of the enhancement layer data or the base layer image. The corresponding full range is 0-1023 for 10-bit.

[0379] S1430: The encoder transmits the code stream to the decoder.

[0380] Corresponding to S1430, the decoder executes S1440. At S1440, the decoder receives a bitstream and obtains metadata from the bitstream.

[0381] S1450: The decoder determines the target data according to the conversion information carried in the metadata.

[0382] The metadata is metadata of the first image, the target data includes base layer data of the first image and / or enhancement layer data of the first image; or, the target data includes the second image.

[0383] The decoder determines the target data according to the conversion position information of the conversion information carried in the metadata.

[0384] (1) If the conversion position information indicates that the conversion occurs after the second picture is obtained, the decoder determines that the target data is the second picture.

[0385] (2) If the conversion position information indicates that the conversion occurs before the second image is obtained, the decoder determines that the target data is at least one of base layer data and enhancement layer data. In this case, the decoder further determines whether the conversion target is the base layer data, the enhancement layer data, or both the base layer data and the enhancement layer data based on the conversion target information contained in the metadata.

[0386] S1460: The decoder converts the target data according to the conversion information.

[0387] Depending on the target data, the process of converting the target data by the decoder is different, which is described below in different cases.

[0388] In the first possible scenario, the target data is base layer data.

[0389] Figure 15 is a schematic diagram of the first conversion process provided by this application. As shown in Figure 15, the decoder converts the base layer data based on the conversion information in the metadata to obtain the converted base layer data. This conversion may include, but is not limited to, color compensation, color gamut conversion, chroma format conversion, and color mode conversion. The decoder converts the color gamut information, chroma format, and color mode of the base layer data based on the color gamut information, chroma format, and color mode carried in the metadata during the combined processing of the base layer data and the enhancement layer data. The following describes the decoder's conversion process for the base layer data.

[0390] (1) For color compensation.

[0391] The decoder can use a compensation matrix, compensation values, etc. to perform color compensation on the base layer data to obtain compensated base layer data. The compensation matrix, compensation values, etc. can be preset or included in metadata, and this application is not limited thereto. For example, the decoder uses the compensation values ​​included in the metadata to perform color compensation on the base layer data to obtain the color-compensated base layer data.

[0392] (2) Regarding color gamut conversion.

[0393] The color gamut information of the base layer data is referred to as the first color gamut information. The color gamut information obtained when the base layer data is combined with the enhancement layer data is referred to as the second color gamut information. Color gamut information can include BT.2020, DCI-P3, Rec.709, NTSC, Rec.2020, and so on. Depending on whether the first and second color gamut information are the same, the decoder processes the color gamut information of the base layer data differently, as described below.

[0394] ① The first color gamut information is the same as the second color gamut information.

[0395] In this case, the decoder may not convert the color gamut information of the base layer data, or may perform an identity transformation on the color gamut information of the base layer data. The method by which the decoder performs an identity transformation on the color gamut information of the base layer data may be carried by metadata or may be preset.

[0396] ② The first color gamut information is different from the second color gamut information.

[0397] In this case, the decoder converts the color gamut information of the base layer data (i.e., the first color gamut information) into the color gamut information during the process of combining the base layer data with the enhancement layer data (i.e., the second color gamut information) based on the color gamut information of the base layer data (or the first color gamut information) and the color gamut information during the process of combining the base layer data with the enhancement layer data (or the second color gamut information). The second color gamut information can be preset color gamut information or color gamut information carried in the metadata. The decoder can use a variety of methods to convert the color gamut information of the base layer data from the first color gamut information to the second color gamut information, such as a conversion matrix, a conversion function, and the like. The conversion matrix or conversion function can be a default one or can be carried in the metadata, and this application is not limited to this. Exemplarily, the decoder uses the 3*3 conversion matrix carried in the metadata to convert the color gamut information of the base layer data from the first color gamut information to the second color gamut information.

[0398] (3) For chroma format conversion.

[0399] The chroma format of the base layer data is the first chroma format, and the chroma format used in the combined processing of the base layer data and the enhancement layer data is the second chroma format. Chroma formats can include R:G:B, YUV4:2:0, YUV4:4:4, YUV4:2:2, and so on. Depending on whether the first chroma format and the second chroma format are the same, the decoder handles the chroma format of the base layer data differently, as described below.

[0400] ① The first chroma format is the same as the second chroma format.

[0401] In this case, the decoder may not convert the chroma format of the base layer data, or may perform an identical transformation on the chroma format of the base layer data. The method by which the decoder performs an identical transformation on the chroma format of the base layer data may be carried by metadata or may be preset.

[0402] ②The first chroma format is different from the second chroma format.

[0403] The decoder converts the chroma format of the base layer data (i.e., the first chroma format) to the chroma format of the base layer data (i.e., the second chroma format) during the processing of combining the base layer data and the enhancement layer data, based on the chroma format of the base layer data (i.e., the first chroma format) and the chroma format of the base layer data (i.e., the second chroma format) during the processing of combining the base layer data and the enhancement layer data. The second chroma format can be a preset chroma format or a chroma format carried in the metadata.

[0404] The chroma format of the base layer data and the chroma format during the combined processing of the base layer data and the enhancement layer data are different, and the way the decoder converts the chroma format is also different, which are described below.

[0405] Method 1: If the first chroma format is YUV4:2:0 or YUV4:2:2, and the second chroma format is R:G:B, the decoder must perform two chroma format conversions on the base layer data, ensuring that the base layer data's chroma format after these two conversions is the same as the enhancement layer data's chroma format, R:G:B. The decoder's first chroma format conversion on the base layer data is to convert the base layer data's chroma format from YUV4:2:0 or YUV4:2:2 to YUV4:4:4. The decoder's second chroma format conversion on the base layer data is to convert the base layer data's chroma format from YUV4:4:4 to R:G:B. The decoder can convert the base layer data's chroma format from YUV4:2:0 or YUV4:2:2 to YUV4:4:4 using various methods, such as upsampling the base layer data from YUV4:2:0 or YUV4:2:2 to YUV4:4:4. The decoder can convert the chroma format of the base layer data from YUV4:4:4 to R:G:B in various ways, such as a conversion matrix, a conversion function, etc. The conversion matrix, conversion function, etc. can be preset or carried in metadata, and this application does not limit this.

[0406] In Method 2, if the first chroma format is YUV4:4:4 and the second chroma format is R:G:B, the decoder performs a single chroma format conversion on the base layer data, making the converted base layer data's chroma format the same as the enhancement layer data's chroma format, R:G:B. The decoder can use various methods to convert the base layer data's chroma format from YUV4:4:4 to R:G:B, such as using a conversion matrix or a conversion function. The conversion process in this case is the same as in Method 1.

[0407] In method 3, if the first chroma format is R:G:B and the second chroma format is YUV4:2:0 or YUV4:2:2, the decoder must perform two chroma format conversions on the base layer data, so that the chroma format of the base layer data after the two chroma conversions is the same as the chroma format of the enhancement layer data, which is YUV4:2:0 or YUV4:2:2. The decoder performs the first chroma format conversion on the base layer data: converting the chroma format of the base layer data from R:G:B to YUV4:4:4. The decoder performs the second chroma format conversion on the base layer data: converting the chroma format of the base layer data obtained from the first conversion from YUV4:4:4 to YUV4:2:0 or YUV4:2:2. The decoder can use various methods to convert the chroma format of the base layer data from R:G:B to YUV4:4:4, such as conversion matrices, conversion functions, etc. The conversion process in this case is the inverse of the conversion process in method 1. The decoder can convert the chroma format of the base layer data from YUV4:4:4 to YUV4:2:0 or YUV4:2:2 in a variety of ways, such as converting the chroma format of the base layer data from YUV4:4:4 to YUV4:2:0 or YUV4:2:2 by downsampling.

[0408] In Method 4, if the first chroma format is R:G:B and the second chroma format is YUV4:4:4, the decoder performs a single chroma format conversion on the base layer data, making the converted chroma format of the base layer data the same as the chroma format of the enhancement layer data, which is YUV4:4:4. The decoder can use various methods to convert the chroma format of the base layer data from R:G:B to YUV4:4:4, such as using a conversion matrix or a conversion function. The conversion process in this case is the inverse of the conversion process in Method 1.

[0409] (4) Regarding color mode conversion.

[0410] The color mode of the base layer data is the first color mode, and the color mode used when combining the base layer data with the enhancement layer data is the second color mode. Color modes can include RGB, YUV, Lab, HSV, RGBA, CMYK, and so on. Depending on whether the first and second color modes are the same, the decoder handles the color mode of the base layer data differently, as described below.

[0411] ①The first color mode is the same as the second color mode.

[0412] In this case, the decoder may not convert the color mode of the base layer data.

[0413] ②The first color mode is different from the second color mode.

[0414] The decoder converts the color mode of the base layer data (i.e., the first color mode) to the color mode in the process of combining the base layer data and the enhancement layer data (i.e., the second color mode) based on the color mode of the base layer data (or the first color mode) and the color mode in the process of combining the base layer data and the enhancement layer data (or the second color mode). The second color mode can be a preset color mode or a color mode carried in the metadata. The decoder can use a variety of methods to convert the color mode of the base layer data from the first color mode to the second color mode, such as a conversion matrix, a conversion function, etc., which are not limited in this application. Exemplarily, the decoder uses the 3*3 conversion matrix carried in the metadata to convert the color mode of the base layer data from the first color mode to the second color mode.

[0415] In the second possible scenario, the target data is enhancement layer data.

[0416] FIG16 is a schematic diagram of the second conversion process provided by the present application. As shown in FIG16 , the decoder converts the enhancement layer data according to the conversion information in the metadata to obtain the converted enhancement layer data. This conversion may include, but is not limited to, color compensation, color gamut conversion, chroma format conversion, and color mode conversion. The methods by which the decoder performs color compensation, color gamut conversion, chroma format conversion, and color mode conversion on the enhancement layer data are the same as the methods for color compensation, color gamut conversion, chroma format conversion, and color mode conversion described in the first possible scenario above. For relevant descriptions, please refer to the above and will not be repeated here.

[0417] In a third possible scenario, the target data is base layer data and enhancement layer data.

[0418] FIG17 is a schematic diagram of the third conversion process provided by this application. As shown in FIG17 , the decoder simultaneously converts the base layer data and the enhancement layer data according to the conversion information in the metadata, thereby obtaining converted base layer data and converted enhancement layer data. Regarding the method by which the decoder converts the base layer data, please refer to the description of the first possible scenario above. Regarding the method by which the decoder converts the enhancement layer data, please refer to the description of the second possible scenario above, and no further details will be given here.

[0419] In a fourth possible scenario, the target data is the second image.

[0420] Figure 18 is a schematic diagram of the fourth conversion process provided by the present application. As shown in Figure 18, the decoder converts the second image according to the conversion information carried in the metadata to obtain the converted second image. The conversion may include, but is not limited to: color compensation, color gamut conversion, chroma format conversion and color mode conversion. Different from the above: the decoder converts the color information, color gamut conversion, chroma format conversion and color mode conversion of the second image according to the color information, color gamut information, chroma format and color mode used in the display process carried in the metadata. The method by which the decoder converts the color information, color gamut information, chroma format and color mode of the second image is the same as the method of color compensation, color gamut conversion, chroma format conversion and color mode conversion in the first possible scenario above. Please refer to the above for the relevant description and will not be repeated here.

[0421] In some possible scenarios, the decoder may also map at least one of the base layer data, the converted base layer data, the enhanced layer data, the converted enhanced layer data, the second image, and the converted second image. The decoder may process at least one of the base layer data, the converted base layer data, the enhanced layer data, the converted enhanced layer data, the second image, and the converted second image in a variety of ways. The processing method may be preset or carried in metadata, and this application is not limited to this. The processing method may include but is not limited to: normalization, inverse normalization, adding an offset, reducing an offset, multiplying by a coefficient, multiplying by a coefficient, and the like. Taking the decoder processing data and obtaining the processed dataAfter as an example, the following processing methods are explained. The data is used to indicate at least one of the base layer data, the converted base layer data, the enhanced layer data, the converted enhanced layer data, the second image, and the converted second image.

[0422] Method ①: Identity transformation. The decoder can use formula (12) to perform identity transformation on data. dataAfter[i]=data[i] Formula (12)

[0423] Where dataAfter[i] is the image after identity transformation and data[i] is the image before transformation.

[0424] Method ②: If the metadata carries the upper limit THH and lower limit THL of data, the decoder can use formula (13) to process data. dataAfter[i]=data[i]*THH+(A-data[i])*THL Formula (13)

[0425] Where dataAfter[i] is the result of processing. data[i] is the image before processing. A is the maximum value of stored data. When data is normalized to between 0 and 1.0, A is 1.0.

[0426] Method ③: If the metadata carries the upper limit THH and lower limit THL of data, data can be nonlinearly coded data such as PQ / HLG / LOG / Gamma. First, data[i] is converted to the linear domain, and then formula (13) is used to process the data[i] converted to the linear domain.

[0427] Method ④: If the metadata carries the lower limit THL of data, the decoder can use formula (14) to process data. dataAfter[i]=data[i]+THL Formula (14)

[0428] Among them, dataAfter[i] is the result obtained by processing, and data[i] is the image before processing.

[0429] Method ⑤: If the metadata carries the upper limit THH of data, the decoder can use formula (15) to process data. dataAfter[i]=THH+A-data[i] Formula (15)

[0430] Where dataAfter[i] is the result of processing. data[i] is the image before processing. A is the maximum value of stored data. When data is normalized to between 0 and 1.0, A is 1.0.

[0431] Method ⑥: If the metadata contains processing parameters, the decoder can obtain the processing method TME() based on the parameters and process data using formula (16). dataAfter[i]=TME(data[i]) Formula (16)

[0432] Where dataAfter[i] is the result of processing. data[i] is the image before processing. The processing method TME() can have various forms, such as sigmoid, cubic spline, gamma, straight line, piecewise curve, or its inverse function, etc., which are not limited in this application. For example, the processing method TME() is the curve represented by formula (17) or its inverse curve.

[0433] Wherein, L and L' can be normalized optical signals or electrical signals, a, b, p, and m are all photoelectric transfer coefficients, with a equal to 1.12762, b equal to -0.12762, p equal to 2.3, m equal to 0.14, and n is a set parameter. The processing method can also be a transformation in the form of a neural network. In this case, the decoder can construct a neural network based on historical data (such as historical enhancement layer data, historical converted enhancement layer data, historical base layer data, historical converted base layer data, etc.), and use the neural network to obtain rules for processing the enhancement layer data, converted enhancement layer data, base layer data, converted base layer data, etc.

[0434] Method 7: If the metadata contains processing parameters, the decoder can obtain the processing method TME() based on the parameters and process data using formula (18). dataAfter[i]=TME()*data[i] Formula (18)

[0435] Among them, dataAfter[i] is the result obtained by processing, and data[i] is the image before processing.

[0436] In the above methods ① to ⑦, i can be a coordinate point. For example, for any coordinate point A(x, y) in a two-dimensional image, i is the coordinate point of coordinate point A in the one-dimensional image after the two-dimensional image is transformed into a one-dimensional image.

[0437] In one possible scenario, if the target data includes base layer data and enhancement layer data, the decoder may further merge the channels of the base layer data and the enhancement layer data according to the channel merging information. The channel merging information is used to indicate how the channels of the base layer data are combined with the channels of the enhancement layer data. The base layer data may have multiple channels. The enhancement layer data may have a single channel or multiple channels. The channel merging information may be preset or carried in metadata. The base layer data and the enhancement layer data may refer to images directly obtained by the decoder parsing the code stream, or images obtained by the decoder after conversion using conversion information, or images obtained by the decoder after processing the images obtained by parsing the code stream. This application does not limit this.

[0438] Different color modes may result in different numbers of channels in an image. For example, the YUV color mode contains three channels: Y, U, and V; the grayscale color mode contains only one channel; and the CMYK color mode contains four channels. The base layer data and enhancement layer data contain different numbers of channels, and the decoder combines the base layer and enhancement layer data in different ways, as described below.

[0439] Case ①: both the base layer data and the enhancement layer data are single-channel.

[0440] In this case, the decoder can merge the channels of the base layer data and the channels of the enhancement layer data according to the channel merging information described in the above case 1, such as directly adding or multiplying the channels of the base layer data and the channels of the enhancement layer data.

[0441] Case ②: the base layer data includes multiple channels, and the enhancement layer data includes a single channel.

[0442] In this case, the decoder can merge the channels of the base layer data and the channels of the enhancement layer data according to the channel merging information described in the above case 2. Depending on whether the color mode of the base layer data is the luma and chroma separation mode, the decoder merges the channels of the base layer data and the channels of the enhancement layer data in different ways, which are explained below.

[0443] 1 The color mode for the base layer data is a color mode in which luminance and chrominance are not separated.

[0444] In this case, the decoder merges the single channel of the enhancement layer data into each channel of the base layer data according to the channel merging information described in (1) of the above scenario 2 to obtain a channel-merged image. For example, the color mode of the base layer data is RGB, and the enhancement layer data is a single-channel image C. In this case, the decoder performs R+C, G+C, B+C, or R*C, G*C, B*C to merge the channels of the base layer data with the channels of the enhancement layer data.

[0445] 2 The color mode for the base layer data is the case where the luminance and chrominance separation mode is used.

[0446] In this case, the decoder can merge the single channel of the enhancement layer data with the luminance channel of the base layer data according to the channel merging information described in (2) of the above case 2 to obtain an image after the merged channels. For example, the base layer data is a three-channel image with separated luminance and chrominance (such as YUV, Lab, HSV, etc.), and the enhancement layer data is a single-channel image C. The decoder merges the base layer data and the enhancement layer data according to the channel merging information described in ① of (2). In this case, the decoder executes Y*C, U*C, V*C or Y+C, U+C, V+C to achieve the merging of the channels of the base layer data and the channels of the enhancement layer data. The decoder merges the base layer data and the enhancement layer data according to the channel merging information described in ② of (2). In this case, the decoder executes Y*C, U*1, V*1 or Y+C, U+0, V+0 to achieve the merging of the channels of the base layer data and the channels of the enhancement layer data.

[0447] Case ③: both the base layer data and the enhancement layer data are multi-channel.

[0448] In this case, depending on whether the color modes of the base layer data and the enhancement layer data are the same, the decoder combines the channels of the base layer data and the channels of the enhancement layer data in different ways, which are described below.

[0449] 1 For the case where the color mode of the base layer data and the enhancement layer data is the same.

[0450] In this case, the decoder can merge the corresponding channels of the base layer data and the enhancement layer data according to the channel merging information described in (1) of the above case 3. If the base layer data and the enhancement layer data are both in RGB mode, the decoder performs R*R, G*G, B*B or R+R, G+G, B+B on the base layer data and the enhancement layer data to merge the channels of the base layer data and the channels of the enhancement layer data. If the base layer data and the enhancement layer data are both in YUV, the decoder performs Y*Y, U*U, V*V or Y+Y, U+U, V+V on the base layer data and the enhancement layer data to merge the channels of the base layer data and the channels of the enhancement layer data.

[0451] 2 For the case where one of the base layer data and the enhancement layer data is in a color mode with separated luminance and chrominance, and the other is not in a color mode with separated luminance and chrominance, and the number of channels is the same.

[0452] In this case, the decoder needs to convert the color mode of at least one of the base layer data and the enhancement layer data, and perform channel merging on the converted data using the merging method described above. The decoder can determine to convert the color mode of the base layer data or the color mode of the enhancement layer data or simultaneously convert the color mode of the base layer data and the color mode of the enhancement layer data based on the color mode in the combined processing of the base layer data and the enhancement layer data carried in the metadata. For example, the base layer data is in RGB mode and the enhancement layer data is in YUV mode. The color mode in the combined processing of the base layer data and the enhancement layer data carried in the metadata is RGB. In this case, the decoder can convert the color mode of the enhancement layer data from YUV to RGB, and the decoder merges the channels of the two converted data using the method described in 1 above.

[0453] 3. For the case where one of the base layer data and the enhancement layer data is in a color mode with separated luminance and chrominance, and the other is not in a color mode with separated luminance and chrominance, and the number of channels of the base layer data and the enhancement layer data are different.

[0454] In this case, the decoder needs to convert the color mode of at least one of the base layer data and the enhancement layer data, and merge the corresponding channels of the converted data. The decoder can determine whether to convert the color mode of the base layer data or the enhancement layer data, or simultaneously convert the color mode of the base layer data and the enhancement layer data based on the color mode of the combined processing of the base layer data and the enhancement layer data carried in the metadata. For example, the base layer data is in RGBA mode and the enhancement layer data is in YUV mode. The color mode of the combined processing of the base layer data and the enhancement layer data carried in the metadata is RGB. In this case, the decoder converts the enhancement layer data from YUV to RGB, and performs R*R, G*G, B*B or R+R, G+G, B+B on the base layer data and the converted enhancement layer data, thereby merging the channels of the base layer data and the channels of the enhancement layer data.

[0455] 4 For the case where both the base layer data and the enhancement layer data are not in a color mode with separated luminance and chrominance and have different numbers of channels.

[0456] In this case, the decoder directly merges the corresponding channels of the base layer data and the enhancement layer data, merging the base layer channels with the enhancement layer channels. For example, if the base layer data is in RGBA mode and the enhancement layer data is in RGB mode, the decoder performs the following operations on the base layer data: R*R, G*G, B*B, or R+R, G+G, B+B, to merge the base layer channels with the enhancement layer channels.

[0457] In the present application, the encoder obtains the base layer data, enhancement layer data and metadata of the first image, and encodes the base layer data, enhancement layer data and metadata to obtain a code stream. The decoder receives and parses the code stream to obtain the base layer data, enhancement layer data and metadata of the first image. The decoder determines and converts the target data (i.e., the base layer data, enhancement layer data or the image obtained by combining the base layer data and the enhancement layer data) based on the conversion information carried by the metadata. In this way, the decoder determines and converts the target data based on the conversion information carried in the metadata, so that the conversion of the target data matches the target data. The decoder converts the target data using the conversion information that matches the target data, so that the dynamic range of the converted image can not only be more adaptable to the dynamic range supported by the display device, but also improve the display effect.

[0458] In this application, the first base layer data may also be referred to as the base layer data of the first image. The first enhancement layer data may also be referred to as the enhancement layer data of the first image. The first metadata may also be referred to as the metadata or metadata of the first image. The second image may also be referred to as the new HDR image.

[0459] An embodiment of the present application also provides a method for processing a signal, comprising: obtaining first base layer data, first enhancement layer data, and first metadata. The first base layer data is base layer data of a first image, the first metadata is metadata of the first image, and the first enhancement layer data is metadata of the first image. Target data is determined based on conversion information carried in the first metadata. The target data is the first base layer data and / or the first enhancement layer data, or the target data is a second image. The second image is an image determined based on the first base layer data and the first enhancement layer data. The target data is converted based on the conversion information.

[0460] In one possible scenario, the first image and the second image are HDR or SDR images, wherein the first metadata may be preset or carried in the bitstream.

[0461] In one possible scenario, the conversion information is conversion instruction information, and the conversion instruction information includes conversion position information. When the conversion position is before the second image is determined, the conversion instruction information further includes conversion object information, and the conversion object information is used to indicate that the target data includes:

[0462] the first base layer data; or

[0463] the first enhancement layer data; or

[0464] The first base layer data and the first enhancement layer data.

[0465] In a possible scenario, when the target data includes the first base layer data, the conversion scheme information includes: a base layer data conversion scheme, where the base layer data conversion scheme is used to indicate performing at least one of the following conversions on the first base layer data:

[0466] Color gamut conversion;

[0467] Color compensation; or

[0468] Color space conversion.

[0469] In a possible scenario, the base layer data conversion scheme is further used to indicate conversion parameters used when converting the first base layer data.

[0470] In a possible scenario, when the base layer data conversion scheme indicates that the color compensation is performed on the first base layer data, the conversion parameter includes any one of the following parameters:

[0471] Color compensation matrix;

[0472] a conversion function; or

[0473] Compensation value and compensation method.

[0474] In one possible scenario, the conversion information is conversion indication information, and the conversion indication information includes conversion scheme information. When the target data includes the first enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, where the enhancement layer data conversion scheme is used to indicate performing at least one of the following conversions on the first enhancement layer data:

[0475] Color space conversion;

[0476] Channel conversion, where the channel conversion is used to restore the channels of the first enhancement layer data to the original channels.

[0477] In a possible scenario, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the first enhancement layer data.

[0478] In a possible scenario, when the target data is the second image, the conversion scheme information includes: an image conversion scheme, where the image conversion scheme is used to indicate performing at least one of the following conversions on the second image:

[0479] Color gamut conversion; or

[0480] Color compensation.

[0481] An embodiment of the present application further provides an encoding method, comprising: determining first base layer data, first enhancement layer data, and first metadata based on a first image acquired, wherein the first metadata carries conversion information, and encoding the first base layer data, the first enhancement layer data, and the first metadata to obtain a bitstream.

[0482] In one possible scenario, the conversion information is conversion indication information, which includes conversion location information and conversion scheme information, wherein the conversion location information is used to indicate the location where the conversion scheme occurs, and the conversion scheme information is used to indicate the conversion scheme. The first metadata may be preset or carried in the bitstream.

[0483] In a possible scenario, the conversion occurs before or after a new HDR image is synthesized based on the received code stream.

[0484] In a possible scenario, when the location where the conversion occurs is before synthesizing a new HDR image, the conversion indication information further includes conversion object information, where the conversion object information is used to indicate target data on which the conversion scheme needs to be executed, and the target data includes:

[0485] Base layer data; or

[0486] Enhancement layer data; or

[0487] The base layer data and the enhancement layer data.

[0488] In a possible scenario, when the target data includes the base layer data, the conversion scheme information includes: a base layer data conversion scheme, where the base layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the base layer data:

[0489] Color gamut conversion; or

[0490] Color compensation; or

[0491] Color space conversion.

[0492] In a possible scenario, the base layer data conversion scheme is further used to indicate conversion parameters used when performing conversion.

[0493] In a possible scenario, when the base layer data conversion scheme indicates to perform the color compensation, the conversion parameter includes any one of the following parameters:

[0494] a color compensation matrix; or

[0495] a conversion function; or

[0496] Compensation value and compensation method.

[0497] In a possible scenario, when the target data includes the enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, where the enhancement layer data conversion scheme is used to indicate performing at least one of the following conversions on the enhancement layer data:

[0498] Color space conversion;

[0499] Channel conversion, where the channel conversion is used to restore the channels of the enhancement layer data to the original channels.

[0500] In a possible scenario, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the first enhancement layer data.

[0501] In a possible scenario, when the conversion scheme occurs after the new HDR image is synthesized, the conversion scheme information includes: an image conversion scheme, where the image conversion scheme is used to indicate that at least one of the following conversions is performed on the new HDR image:

[0502] Color gamut conversion; or

[0503] Color compensation.

[0504] In one possible scenario, the conversion information includes at least one or more of the following:

[0505] Color mode information of the base layer of the first image; and / or

[0506] Color mode information of the enhancement data of the first image; and / or

[0507] Color mode information for transforming and processing the base layer of the first image and the enhancement data. When the color space model of the base layer image is a first color mode / space, and the color space model of the process of combining the base layer with the enhancement image is a second color mode / space, the color mode information for transforming and processing the base layer of the first image and the enhancement data includes a color mode / space model conversion method, and the first color mode / space is different from the second color mode / space.

[0508] The conversion information includes at least one or more of the following:

[0509] Color mode information of the base layer of the first image; and / or

[0510] Color mode information of the enhancement layer image of the first image; and / or

[0511] Color mode information for transforming and processing the base layer and enhancement layer images of the first image. When the color space model of the base layer image is a first color mode / space, and the color space model of the image combined with the enhancement image for processing is a second color mode / space, the color mode information for transforming and processing the base layer and enhancement data of the first image includes a color mode / space model conversion method, and the first color mode / space is different from the second color mode / space.

[0512] The first color mode / space and the second color mode / space are one of the following:

[0513] RGB, YUV, YCbCr, Lab, HSV, or ICtCp.

[0514] This makes the conversion more flexible, and metadata can be adapted to multiple formats, which can facilitate more efficient and reasonable encoding.

[0515] In one possible scenario, the conversion information includes at least one or more of the following:

[0516] Chroma format information of the base layer of the first image; and / or

[0517] Chroma format information of the enhancement data of the first image; and / or

[0518] Chroma format information for conversion and processing of a base layer of a first image and enhancement data. When the chroma format information of the base layer is first chroma format information, and the chroma format information for processing the base layer and the enhancement data is second chroma format information, the conversion information includes the chroma format information for conversion and processing of the base layer of the first image and the enhancement data, and the first chroma format information is different from the second chroma format information.

[0519] The conversion information includes at least one or more of the following:

[0520] Chroma format information of the base layer of the first image; and / or

[0521] Chroma format information of the enhancement layer picture of the first picture; and / or

[0522] Chroma format information for conversion and processing of a base layer image and an enhancement layer image of a first image. When the chroma format information of the base layer is first chroma format information, and the chroma format information for processing the base layer image and the enhancement layer image is second chroma format information, the conversion information includes the chroma format information for conversion and processing of the base layer image and the enhancement layer image of the first image, and the first chroma format information is different from the second chroma format information.

[0523] The first chroma format information and the second chroma format information are one of the following:

[0524] RGB4:4:4, grayscale, YUV4:4:4, YUV4:2:2, YUV4:2:0, two-channel mode.

[0525] This allows for a higher degree of matching of the color gamuts of different display devices, supports color space conversion, and facilitates more efficient and reasonable encoding.

[0526] In one possible scenario, the conversion information includes at least one or more of the following:

[0527] Information on how to combine the color mode multi-channel of the base layer of the first image and the color mode multi-channel or single channel of the enhancement data; or

[0528] The information on how to combine the color mode multi-channel of the base layer of the first image with the color mode multi-channel or single channel of the enhancement layer image provides greater flexibility in conversion, and metadata can be flexibly adapted in various ways.

[0529] This application also provides another embodiment, which is used on the receiving and display side to decode and display all pixels in an image or video in SDR or HDR format. It mainly includes the following processes:

[0530] 1. Obtain base layer images, enhancement layer images / enhancement data, and metadata from the bitstream.

[0531] 1. Acquisition of metadata.

[0532] This information can be obtained from metadata encapsulated in the bitstream, such as SEI in HEVC or VVC, user-defined NAL units or reserved packet units, APP extension information encapsulated in JFIF, or data segments encapsulated in MP4, etc. Metadata information can also be obtained from other preset channels.

[0533] a) Metadata can include color mode information.

[0534] The metadata may include color mode information for the base layer, and / or color mode information for the enhancement data, and / or color mode information for the conversion and processing between the base layer and the enhancement data. The color space model or color mode information for the base layer / enhancement layer refers to whether the base layer or enhancement layer image is one of RGB, YUV, YCbCr, Lab, HSV, or ICtCp.

[0535] Metadata can also include color mode conversion information. Assuming the base layer image color space model is a first color mode / space (e.g., RGB), but the color space model used in the processing of the enhanced image / enhancement data is a second color mode / space (e.g., YUV), a color mode / space model conversion process is required. This process can be performed using a preset conversion method or a color mode conversion method transmitted via metadata.

[0536] b) Metadata can include luma and chroma format information.

[0537] The metadata may include information on the luminance component and chroma component format (chroma subsampling) of the base layer image and the enhancement layer image (e.g., RGB4:4:4, grayscale, YUV4:4:4, YUV4:2:2, YUV4:2:0, two-channel mode), which is used for transformation and processing of the base layer image and / or the enhancement layer image.

[0538] c) The metadata may include information on how to combine the color mode multi-channel of the base layer with the color mode multi-channel or single channel of the enhancement data.

[0539] The metadata includes channel information of the base layer and the enhancement layer, which is used to indicate how to merge the dual-layer multi-channel images.

[0540] 2. Acquisition of basic layer image.

[0541] The base layer image can be used as a source in any format, and the corresponding transfer function can be used during encoding and transmission. The decoding end decodes the image according to the preset information or the relevant information in the metadata to obtain the base layer image. This process includes decoding the received code stream through any decoder (HEVC, JPEG, ProRes, HEIF, AV1F, etc.) to obtain the image data. The image data obtained in this process includes image data of any color space / mode (RGB, YUV, YCbCr, CIELAB, HSV, HSL, ICtCp, etc.). It should be noted that the present invention does not limit the code stream format. In terms of color space, it can be YUV or RGB. In terms of data bit width, it can be 8 bits, 10 bits, or 12 bits, etc. In terms of numerical domain, it can be a linear domain or a non-linear domain such as PQ / HLG / Gamma / log.

[0542] a) Color mode / space conversion of base layer image.

[0543] If the color space model of the base layer image is a first color mode / space (e.g., YUV), but the color space model of the base layer and enhancement layer image / enhancement data combined processing is a second color mode / space (e.g., RGB), a color mode / space model conversion process is required. Based on the base layer image color space model information (preset or transmitted in metadata) and the color space model information of the base layer and enhancement layer image / enhancement data combined processing (preset or transmitted in metadata), the base layer image is transformed / converted to obtain a base layer image in a new format. The transformation / conversion method includes the default method or the description of the conversion process in the metadata, and can use the 3x3 conversion matrix specified in the ITU standard or other methods.

[0544] b) Chroma format conversion of base layer image.

[0545] If the luma and chroma components of the base layer image are in a first chroma format (e.g., YUV4:2:0), and the luma and chroma components of the new base layer image to be processed and converted are in a second chroma format (e.g., RGB4:4:4), the base layer image's chroma format must be converted during the conversion process. This includes chroma format upsampling from YUV4:2:0 to YUV4:4:4, and a second color mode / space model conversion from YUV4:4:4 to RGB4:4:4. Based on the base layer image's chroma format information (preset or transmitted in metadata) and the chroma format information (preset or transmitted in metadata) of the combined processing of the base layer image and the enhanced image, the base layer image is converted to a new base layer format. This conversion process can be performed using a pre-set conversion method or a conversion method transmitted via metadata, and can include chroma upsampling and other transformations.

[0546] c) Other relevant transformations of the base layer image.

[0547] It should be noted that the present invention does not limit the order of the color mode / space model conversion process and the color format conversion process. The present invention does not require that the color mode / space model conversion process and the color format conversion process be performed simultaneously. Only one or both of them may be performed.

[0548] It should be noted that the color mode / space model conversion process and the chroma format conversion process described above can be preceded or followed by corresponding mapping processes described in accordance with presets or metadata. Such mapping is not mandatory in this patent and may include, for example, normalization and denormalization, adding and subtracting offsets, multiplying and dividing by numerical coefficients, etc.

[0549] It should be noted that the color mode / space model conversion process and the chroma format conversion process described above can be preceded or followed by corresponding upsampling / downsampling processes as described in the presets or metadata. After downsampling, a basic image with a new resolution can be obtained.

[0550] It should be noted that the metadata may include processing information for the base layer and enhancement layer, or preset processing information may be used. The base layer Base can be processed based on the metadata to obtain BaseAfter. The processing scheme can be:

[0551] 1.BaseAfter[i]=Base[i].

[0552] 2. The metadata contains the first preset value THH and the second preset value THL of the base layer data, using BaseAfter[i]=Base[i]*THH+(A-Base[i])*THL, where A is the maximum value stored in Base. When Base is normalized to between 0 and 1.0, A is 1.0.

[0553] 3. The metadata includes a first preset value THH and a second preset value THL of the base layer data, where the base layer data can be non-linear coding data such as PQ / HLG / LOG / Gamma. First, Base[i] is converted to the linear domain, and then Base After[i] = Base[i]*THH+(A-Base[i])*THL is used, where A is the maximum value stored in Base. When Base is normalized to between 0 and 1.0, A is 1.0.

[0554] 4. The metadata contains the preset value THL of the base layer data, using BaseAfter[i]=Base[i]+THL.

[0555] 5. The metadata contains the preset value THH of the base layer data, using BaseAfter[i]=THH+A-Base[i], where A is the maximum value stored in Base. When Base is normalized to between 0 and 1.0, A is 1.0.

[0556] 6. The metadata contains the parameters of the mapping relationship, and TME() is obtained according to the parameters, BaseAfter[i]=TMB(Base[i]).

[0557] 7. The metadata contains the parameters of the mapping relationship, and TME() is obtained according to the parameters, BaseAfter[i]=TME()*Base[i].

[0558] The mapping relationship has various forms, including Sigmoid, cubic spline, gamma, straight line, segmented curve, etc. or their inverse function forms, which are not limited in the present invention. The curve represented by the following formula (19) or its inverse curve can be used:

[0559] It should be noted that L and L' can be normalized optical or electrical signals, and this is not a limitation in the present invention. Alternatively, a transformation can be performed in the form of a neural network: obtaining information about the type of neural network, such as a transformer or convolutional neural network; constructing a neural network based on the network information in the enhanced data; and processing the enhanced data using the neural network. It should be noted that this can be global tone mapping or local tone mapping, and this is not a limitation in the present invention.

[0560] 3. Acquisition of enhancement layer image.

[0561] The enhancement layer image serves as a bridge between the base layer and the final merged image. It can adopt any format. The corresponding transfer function can be used during encoding and transmission. The decoding end decodes according to the preset information or the relevant information in the metadata to obtain the enhancement layer image. This process includes decoding the received code stream through any decoder (HEVC, JPEG, ProRes, HEIF, AV1F, etc.) to obtain the image data. The image data obtained in this process includes image data in any color space mode (RGB, YUV, YCbCr, CIELAB, HSV, HSL, ICtCp, etc.). It should be noted that the present invention does not limit the code stream format. In terms of color space, it can be YUV or RGB. In terms of data bit width, it can be 8 bits, 10 bits, or 12 bits, etc. In terms of numerical domain, it can be a linear domain or a non-linear domain such as PQ / HLG / Gamma / log.

[0562] a) The first color mode / space conversion method of the enhanced image.

[0563] If the decoded base layer image is in the first color space (e.g., RGB), the enhanced layer image is in the second color space (e.g., YUV), and the synthesized new dynamic range image is required to be in the first color space, the enhanced layer image must first be converted to the first color space. Transformation methods include the default matrix method or metadata transmission (e.g., a 3x3 transformation matrix).

[0564] (No transformation) If the base layer image obtained after decoding is in the first color space (such as RGB), the enhancement layer image is also in the first color space (RGB), and the new dynamic range image after synthesis is required to be in the first color space, no transformation is performed or the default identity transformation is used.

[0565] If the decoded base layer image is in the first color space with Chroma and Luma separation (such as YUV or CIELAB), and the enhanced layer image is also in the first color space (YUV), and the synthesized new dynamic range image is required to be in the second color space (RGB), the enhanced layer image must first be converted to the first color space. Conversion methods include the default conversion method or the conversion method transmitted through metadata.

[0566] If the base layer image obtained after decoding is in the first color space (such as HSV), the enhancement layer image is in the second color space (such as YUV), and the new dynamic range image after synthesis is required to be in the third color space (such as RGB), it is necessary to perform a preset transformation when synthesizing the new dynamic range image or perform a color space transformation based on metadata.

[0567] b) A second color space / mode conversion method for enhancing the image.

[0568] If the base layer image obtained after decoding is in the first color space (such as RGB), and the base layer and enhancement layer image / enhancement data are processed in the second color space (such as YUV), the enhancement layer image needs to be converted to the second color space first. According to the enhancement layer image / enhancement data color space model information (preset or transmitted in metadata) and the color space model information of the base layer and enhancement layer image processing (preset or transmitted in metadata), the enhancement layer image / enhancement data is transformed / converted to obtain a new format of the enhancement layer image / enhancement data. The transformation method includes the default method or the description of the conversion process transmitted in the metadata, which can be the 3x3 matrix method specified by ITU.

[0569] (No Transformation) If the base layer image obtained after decoding is in the first color space (such as RGB), and the base layer and enhanced image are processed in the first color space, no transformation is performed or the default identity transformation is used.

[0570] c) Enhanced image chroma format conversion.

[0571] If the luma and chroma components of the enhancement layer image are in a first chroma format (e.g., YUV4:2:0), and the luma and chroma components of the new enhancement layer image to be processed and converted are in a second chroma format (e.g., YUV4:4:4), or if the chroma format information from the combined base layer and enhancement image processing is in a third chroma format (e.g., RGB4:4:4), the enhancement image must be converted during the conversion process. This includes: first, upsampling conversion from YUV4:2:0 to YUV4:4:4, and second, color space model conversion from YUV4:4:4 to RGB4:4:4. Based on the chroma format information of the enhancement layer image (preset or transmitted in metadata) and the chroma format information from the combined base layer and enhancement image processing (preset or transmitted in metadata), the enhancement image is converted to a new base layer image format. This conversion process can be performed according to a predefined conversion method or a conversion method transmitted via metadata.

[0572] d) Other related transformations of enhancement layer images / enhancement data.

[0573] It should be noted that the present invention does not limit the order of the color mode / space model conversion process and the color format conversion process. The present invention does not require that the color mode / space model conversion process and the color format conversion process be performed simultaneously. Only one or both of them may be performed.

[0574] It should be noted that the color mode / space model conversion process and the chroma format conversion process described above can be preceded or followed by corresponding mapping processes described in accordance with presets or metadata. Such mapping is not mandatory in this patent and may include, for example, normalization and denormalization, adding and subtracting offsets, multiplying and dividing by numerical coefficients, and upsampling and downsampling images.

[0575] It should be noted that the color mode / space model conversion process and the chroma format conversion process described above can be preceded or followed by corresponding upsampling / downsampling processes as described in the presets or metadata. After downsampling, an enhanced image with a new resolution can be obtained.

[0576] It should be noted that metadata can contain processing information for the base layer and enhancement layer, or preset processing information can be used. Taking the enhancement layer as an example, EnhanceAfter can be obtained by processing the enhancement layer Enhance based on metadata. The processing scheme can be:

[0577] 1.EnhanceAfter[i]=Enhance[i].

[0578] 2. The metadata contains the first preset value THH and the second preset value THL of the enhancement layer data, using EnhanceAfter[i]=Enhance[i]*THH+(A-Enhance[i])*THL, where A is the maximum value stored in Enhance. When Enhance is normalized to between 0 and 1.0, A is 1.0.

[0579] 3. The metadata includes a first preset value THH and a second preset value THL of the enhancement layer data, where the enhancement layer data can be non-linear coding data such as PQ / HLG / LOG / Gamma. First, Enhance[i] is converted to the linear domain, and then EnhanceAfter[i] = Enhance[i]*THH+(A-Enhance[i])*THL is used, where A is the maximum value stored in Enhance. When Enhance is normalized to between 0 and 1.0, A is 1.0.

[0580] 4. The metadata contains the preset value THL of the enhancement layer data, and uses EnhanceAfter[i]=Enhance[i]+THL.

[0581] 5. The metadata contains the preset value THH of the enhancement layer data, using EnhanceAfter[i]=THH+A-Enhance[i], where A is the maximum value stored in Enhance. When Enhance is normalized to between 0 and 1.0, A is 1.0.

[0582] 6. The metadata contains the parameters of the mapping relationship, and TME() is obtained according to the parameters, where EnhanceAfter[i]=TMB(Enhance[i]).

[0583] 7. The metadata contains the parameters of the mapping relationship, and TME() is obtained according to the parameters, where EnhanceAfter[i]=TME()*Enhance[i].

[0584] The mapping relationship can take various forms, including sigmoid, cubic spline, gamma, straight line, piecewise curve, etc., or their inverse function forms, which are not limited in the present invention. The curve represented by formula (19) above or its inverse curve can be used, and formula (19) can be expressed again as follows:

[0585] It should be noted that L and L' can be normalized optical or electrical signals, and this is not a limitation in the present invention. Alternatively, a transformation can be performed in the form of a neural network: obtaining information about the type of neural network, such as a transformer or convolutional neural network; constructing a neural network based on the network information in the enhanced data; and processing the enhanced data using the neural network. It should be noted that this can be global tone mapping or local tone mapping, and this is not a limitation in the present invention.

[0586] Second, the base layer image and the enhancement layer image are merged according to preset information or metadata to obtain a reconstructed new dynamic range image.

[0587] According to the previous process, the enhanced layer processing result EnhanceAfter can be obtained, and combined with BaseAfter to obtain the merged result newPictue.

[0588] a) Get the method of merging multiple color channels.

[0589] Based on information transmitted in metadata or preset regarding how to combine the base layer's multi-channel color mode with the enhancement data's multi-channel or single-channel color mode, a method for merging the multiple color channels of the base layer and enhancement layer images / enhancement data is obtained. Both the base layer image and the enhancement layer image / enhancement data are in a specific color mode or color space. Different color modes or color spaces contain different numbers of color channels. For example, the YUV color mode contains three color channels: Y, U, and V; the grayscale color mode contains only one color channel; and the CMYK color mode contains four color channels.

[0590] Assuming that BaseAfter is a single color channel and EnhanceAfter is a single color channel, the merging mode information may include a one-to-one combining mode according to preset merging mode information or merging mode information obtained from metadata.

[0591] Assuming that BaseAfter is a multi-color channel and EnhanceAfter is a single color channel, the merging method information can include one-to-one or many-to-one merging, according to the preset merging method information or the merging method information obtained from metadata. For example:

[0592] Many-to-one: For example, if BaseAfter is a three-channel image (such as RGB) with inseparable luminance and chrominance, and EnhanceAfter is a single-channel C image, then perform R*C, G*C, B*C, or R+C, G+C, B+C, that is, multiply or add each channel separately.

[0593] One-to-one: For example, if BaseAfter is a three-channel image with separated brightness and chrominance (such as YUV, Lab, HSV, etc.), then perform multiplication or addition operations on the brightness channel only, without performing operations on the chrominance channels.

[0594] Assuming that BaseAfter has multiple channels and EnhanceAfter has multiple channels, the merging mode information may include a many-to-many combination mode according to preset merging mode information or merging mode information obtained from metadata.

[0595] Three to two: For example, BaseAfter is a brightness and color separation model (such as Lab, YUV), and EnhanceAfter has two color channels (for example, represented by YC), then the merging operation is performed on the corresponding brightness and color channels, such as baseY*enhenceAfterY, baseU*enhenceAfterC, baseV*enhenceAfterC.

[0596] Three-to-two: For example, BaseAfter is a brightness and color separation model (such as HSV, HIS, etc.), and EnhanceAfter has two color channels (such as YC), then the merging operation is performed on the corresponding brightness (V or I channel, etc.) and color saturation or hue channel, such as baseV*enhenceAfterY, baseS*enhenceAfterC, baseH remains unchanged, or baseV*enhenceAfterY, baseH*enhenceAfterC, baseS remains unchanged.

[0597] Many-to-many: For example, if BaseAfter and EnhanceAfter have the same color space model (such as RGB, YUV), the merging operation is performed on the corresponding channels, such as R*R, G*G, B*B.

[0598] Many-to-many: For example, if the color space model of BaseAfter is RGB, and the color space model of EnhanceAfter is a color space model with separated brightness and chroma (such as YUV, Lab), it is necessary to first convert from the second color space to the first color space (such as YUV to RGB), and then merge the corresponding channels.

[0599] Many-to-many: For example, if the base layer and enhancement layer have multiple channels and the number of channels is greater than three (such as RGBA), a reasonable two-layer image merging operation should be performed based on the channel characteristics and information such as brightness and chromaticity separation. This can be done by only processing the corresponding color channels such as RGB.

[0600] b) Perform merging calculations based on the multi-color channel merging method.

[0601] Solution 1: newPictue[i] = BaseAfter[i]*f(EnhanceAfter[i]), where f(x) is the numerical domain conversion function.

[0602] Solution 2: newPictue[i]=BaseAfter[i]+f(EnhanceAfter[i]), where f(x) is the numerical domain conversion function.

[0603] Solution 3: newPictue[i] = g(BaseAfter[i]*f(EnhanceAfter[i])), where f(x) is the numerical domain conversion function and g(x) is the mapping function.

[0604] Solution 4: newPictue[i]=g(BaseAfter[i]+f(EnhanceAfter[i])), where f(x) is the numerical domain conversion function and g(x) is the mapping function.

[0605] It should be noted that the present invention does not limit the form of f(x) and g(x), that is, it does not limit BaseAfter[i] and EnhanceAfter[i] to any domain, which can be a linear domain, PQ domain, log domain, etc. In addition, the present invention does not limit whether other image processing is performed on the reconstructed new dynamic range image newPictue after obtaining it and before displaying it.

[0606] The present application also provides another embodiment, wherein the base layer is in the first format (first transfer function), and an image in the second format (second transfer function) is obtained through the enhancement layer, and the enhancement layer is the second or third transfer function; or

[0607] The base layer is in the first color space, and the image in the second color space is obtained through the enhancement layer; or

[0608] The base layer is in the first color gamut space, which is changed to the second color gamut space through the 3x3 matrix passed in the metadata or by default, and then a higher dynamic range image in the second color gamut space is obtained through the enhancement layer; or

[0609] The base layer is the first color space YUV, the enhancement layer is the first YUV or the second RGB color space, and after the base layer is converted to the second RGB color space, an image in the second color space is obtained through the enhancement layer.

[0610] In the embodiments of the present application, the symbol * and the symbol × can be interchanged.

[0611] The technical solution of the present application can use a mapping function to process the base image during the process of transforming the base image according to the metadata; can use a mapping function to process the enhanced data during the process of transforming the enhanced data according to the metadata; can perform up and down sampling during the process of transforming the enhanced data according to the metadata, use the interpolation algorithm in the texture sampling algorithm in OPENGL, Vulkan, or Metal for processing, or use an index to select a better algorithm from a set of interpolation algorithms.

[0612] The data processing method provided by this application is described above in conjunction with the accompanying drawings. Below, an encoding device provided by this application is described in conjunction with the accompanying drawings. Figure 19 is a schematic structural diagram of an encoding device provided by this application. The encoding device 1900 includes: a processing module 1910 and an encoding module 1920. The processing module 1910 is used to perform step S1410, and the encoding module 1920 is used to perform step S1420. In some possible examples, the encoding device 1900 may further include a transceiver module 1930 for performing step S1430. The encoding device can be used to implement the functions implemented by the source device 1010 in Figure 10.

[0613] It should be understood that the encoder of the aforementioned embodiment may correspond to the encoding device 1900, and may correspond to the corresponding subject corresponding to the execution of the method embodiment of the present application, and the operations and / or functions of each module in the encoding device 1900 are respectively for implementing the corresponding processes of each method of the embodiment. For the sake of brevity, they will not be repeated here.

[0614] The encoding apparatus provided by this application is described above in conjunction with the accompanying drawings. The encoding device provided by this application is described below in conjunction with the accompanying drawings. Figure 20 is a schematic structural diagram of an encoding device provided by this application. As shown in Figure 20, encoding device 2000 may include a processor 2010 for executing programs or instructions stored in memory 2020. When the programs or instructions stored in memory 2020 are executed, the processor is used to execute the methods described in the aforementioned embodiments of this application. Encoding device 2000 can be used to implement the functions implemented by source device 1010 in Figure 10 above.

[0615] In one possible scenario, the encoding device 2000 may further include a communication interface 2030. FIG20 uses a dotted line to indicate that the communication interface 2030 is optional for the encoding device 2000. When the communication interface 2030 implements the function of the transceiver module 1910, it is used to receive the first image.

[0616] Among them, the number of processors 2010, memories 2020 and communication interfaces 2030 does not constitute a limitation on the embodiments of the present application. During specific implementation, they can be arbitrarily configured according to business requirements.

[0617] In one possible scenario, the memory 2020 is located outside the encoding device 2000 .

[0618] In one possible scenario, encoding device 2000 includes a memory 2020, the memory 2020 being connected to at least one processor 2010, and the memory 2020 storing instructions executable by the at least one processor 2010. Memory 2020 may also be used to store a first image, base layer data of the first image, enhancement layer data of the first image, and metadata for the first image. FIG. 20 uses dashed lines to indicate that memory 2020 is optional for encoding device 2000.

[0619] The processor 2010 and the memory 2020 may be coupled via an interface circuit or integrated together, which is not limited here. The processor 2010 is used to implement the functions of the processing module 1910 in the encoding device described in FIG. 19 above.

[0620] The specific connection medium between the processor 2010, memory 2020, and communication interface 2030 is not limited in the embodiments of the present application. In Figure 20, the processor 2010, memory 2020, and communication interface 2030 are connected via a bus 2040. The bus is represented by a bold line in Figure 20. The connection methods between other components are merely schematic and are not limiting. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, Figure 20 only uses a single bold line, but this does not mean that there is only one bus or only one type of bus.

[0621] The encoding device provided by this application is described above in conjunction with the accompanying drawings. The decoding device provided by this application is described below in conjunction with the accompanying drawings. Figure 21 is a schematic structural diagram of a decoding device provided by this application. As shown in Figure 21, the decoding device includes: a transceiver module 2110, a processing module 2120, and a conversion module 2130. The transceiver module 2110 is used to perform step S1440, the processing module 2120 is used to perform step S1450, and the conversion module 2130 is used to perform step S1460.

[0622] Processing module 2120 is configured to receive a code stream. Processing module 2120 is configured to determine target data based on conversion information carried in the metadata. The metadata is metadata of a first image. The target data includes base layer data of the first image and / or enhancement layer data of the first image, or the target data includes a second image. The second image is obtained based on the base layer data and the enhancement layer data. Conversion module 2130 is configured to convert the target data based on the conversion information.

[0623] It should be understood that the decoder of the aforementioned embodiment may correspond to the decoding device 2100, and may correspond to the corresponding subject corresponding to the execution of the method embodiment of the present application, and the operations and / or functions of each module in the decoding device 2100 are respectively for implementing the corresponding processes of each method of the embodiment. For the sake of brevity, they will not be repeated here.

[0624] The decoding apparatus provided in this application is described above in conjunction with the accompanying drawings. The decoding device provided in this application is described below in conjunction with the accompanying drawings. Figure 22 is a schematic structural diagram of a decoding device provided in this application. As shown in Figure 22, decoding device 2200 includes a processor 2210 and a memory 2220. Processor 2210 is configured to execute programs or instructions stored in memory 2220. When the programs or instructions stored in memory 2220 are executed, processor 2210 is configured to perform the method described in the embodiment of Figure 14.

[0625] In one possible scenario, decoding device 2200 may further include a communication interface 2230. FIG22 uses dashed lines to indicate that communication interface 2230 is optional for decoding device 2200. Communication interface 2230 is used to implement the functionality of transceiver module 2110 in FIG21 above, to execute S1440 in the method embodiment. Processor 2210 is used to implement the functionality of processing module 2120 and conversion module 2130 in FIG21 above, to execute S1450 and S1460 in the method embodiment.

[0626] Among them, the number of processors 2210, memories 2220 and communication interfaces 2230 does not constitute a limitation on the embodiments of the present application. During specific implementation, they can be arbitrarily configured according to business requirements.

[0627] In one possible scenario, the memory 2220 is located outside the decoding device 2200 .

[0628] In one possible scenario, decoding device 2200 includes a memory 2220, which is connected to at least one processor 2210. Memory 2220 stores instructions executable by at least one processor 2210. Memory 2220 is used to store the codestream, as well as base layer data and enhancement layer data of the decoded first image, metadata of the first image, and the like. FIG22 uses dashed lines to indicate that memory 2220 is optional for decoding device 2200.

[0629] The processor 2210 and the memory 2220 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0630] The specific connection medium between the processor 2210, memory 2220, and communication interface 2230 is not limited in the embodiments of the present application. In Figure 22, the processor 2210, memory 2220, and communication interface 2230 are connected via a bus 2240. The bus is represented by a bold line in Figure 22. The connection methods between other components are only for schematic illustration and are not limiting. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, only one bold line is used in Figure 22, but this does not mean that there is only one bus or one type of bus.

[0631] It should be understood that the processors mentioned in the embodiments of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that is implemented by reading software code stored in a memory.

[0632] Exemplarily, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0633] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0634] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0635] It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0636] An embodiment of the present application further provides a computer-readable storage medium, comprising a program or instructions, which, when executed on a computer, enables the encoding and decoding method in the method embodiment described above to be implemented.

[0637] The term "at least one" in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where any of a, b, c, ab, ac, bc, or abc can include a single a, a single b, a single c, and can also include multiple a, multiple b, and multiple c.

[0638] Furthermore, unless otherwise indicated, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish between multiple objects and are not used to define the order, timing, priority, or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria and do not indicate differences in the content, priority, or importance of the two criteria.

[0639] In addition, the terms "including" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules / units is not limited to the listed steps or modules and may also include steps or modules / units that are not listed.

[0640] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0641] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.

[0642] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0643] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0644] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is intended to include such modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the target data according to the conversion information carried in the metadata; wherein the metadata is metadata of a first image, and the target data includes base layer data of the first image and / or enhancement layer data of the first image; or, the target data includes a second image, wherein the second image is obtained according to the base layer data and the enhancement layer data; The target data is converted according to the conversion information.

2. The method according to claim 1, characterized in that The metadata is carried in the bitstream. The method further includes: acquiring the metadata from the code stream.

3. The method according to claim 1 or 2, characterized in that: The conversion information includes conversion position information and conversion scheme information, wherein the conversion position information is used to indicate the position where the conversion occurs, and the conversion scheme information is used to indicate the conversion scheme used when converting the target data. The conversion occurs before or after obtaining the second image.

4. The method according to any one of claims 1 to 3, characterized in that In the case where the conversion occurs before the second image is obtained, the conversion information further includes conversion object information, where the conversion object information is used to indicate a conversion object on which the conversion scheme needs to be executed, and the conversion object includes: Base layer data of the first image; or enhancement layer data of the first image; or, Base layer data of the first image and enhancement layer data of the first image.

5. The method according to claim 4, characterized in that In the case where the conversion object includes the basic layer data, the conversion scheme information includes: a basic layer data conversion scheme, where the basic layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the basic layer data: Color gamut conversion; Chroma format conversion; Color compensation; or Color mode conversion.

6. The method according to claim 5, characterized in that The basic layer data conversion scheme is also used to indicate conversion parameters used when converting the basic layer data.

7. The method according to claim 6, characterized in that The conversion parameters include: compensation matrix; or, a transformation matrix; or, a conversion function; or, Compensation value.

8. The method according to claim 4, characterized in that In the case where the conversion object includes the enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, where the enhancement layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the enhancement layer data: Color mode conversion; Chroma format conversion; or, Channel conversion, where the channel conversion is used to restore the channels of the enhancement layer data to the original channels.

9. The method according to claim 8, characterized in that The enhancement layer data conversion scheme is also used to indicate conversion parameters used when converting the enhancement layer data.

10. The method according to claim 9, characterized in that The conversion parameters include: a transformation matrix; or, a conversion function; or, Channel conversion function.

11. The method according to any one of claims 1 to 10, characterized in that The conversion information includes channel merging information; the channel merging information is used to indicate: a combination scheme of the channel of the basic layer data and the channel of the enhancement layer data.

12. The method according to claim 11, characterized in that The color mode of at least one of the base layer data and the enhancement layer data is a red, green, blue (RGB) mode or a grayscale mode; Alternatively, the color mode of the base layer data is an RGB mode, and the color mode of the enhancement layer data is any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode; Alternatively, the color mode of the base layer data is any one of a YUV mode, a Lab mode, a HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode, and the color mode of the enhancement layer data is an RGB mode; Alternatively, the color mode of the basic layer data and the color mode of the enhanced layer data are any one of: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode.

13. The method according to claim 12, characterized in that In the case where the color modes of the basic layer data and the enhancement layer data are both RGB modes, the channel merging information is used to indicate: an R channel of the basic layer data is merged with an R channel of the enhancement layer data, a G channel of the basic layer data is merged with a G channel of the enhancement layer data, and a B channel of the basic layer data is merged with a B channel of the enhancement layer data; When the color mode of the base layer data is the RGB mode and the color mode of the enhancement layer data is the grayscale mode, the channel merging information is used to indicate: the R channel of the base layer data is merged with the grayscale value of the enhancement layer data, the G channel of the base layer data is merged with the grayscale value of the enhancement layer data, and the B channel of the base layer data is merged with the grayscale value of the enhancement layer data; When the color mode of the base layer data is a grayscale mode and the color mode of the enhancement layer data is an RGB mode, the channel merging information is used to indicate: an R channel of the enhancement layer data is merged with a grayscale value of the base layer data, a G channel of the enhancement layer data is merged with a grayscale value of the base layer data, and a B channel of the enhancement layer data is merged with a grayscale value of the base layer data; In the case where the color modes of the base layer data and the enhancement layer data are both grayscale modes, the channel merging information is used to indicate that the grayscale value of the base layer data and the grayscale value of the enhancement layer data are merged; In the case where the color mode of the base layer data and the color mode of the enhancement layer data are any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode, the channel merging information is used to indicate that the brightness channel of the base layer data and the brightness channel of the enhancement layer data are merged; In the case where the color modes of the base layer data and the enhancement layer data are both any one of the YUV mode, the Lab mode, the HSV mode, the YCbCr mode, and the ICtCp mode, the channel merging information is used to indicate that the brightness channel of the base layer data and the brightness channel of the enhancement layer data are merged; When the color mode of the base layer data is a grayscale mode and the color mode of the enhancement layer data is any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, and an ICtCp mode, the channel merging information is used to indicate that a channel of the base layer data and a brightness channel of the enhancement layer data are merged; When the color mode of the basic layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, and the color mode of the enhanced layer data is a grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the channel of the enhanced layer data are merged.

14. The method according to claim 5 or 8, characterized in that: The color mode conversion includes: Convert the color mode of the basic layer data so that the converted color mode of the basic layer data is the same as the color mode of the enhancement layer data, or convert the color mode of the enhancement layer data so that the converted color mode of the enhancement layer data is the same as the color mode of the basic layer data.

15. The method according to any one of claims 1 to 3, characterized in that: In the case where the conversion occurs after the second image is obtained, the conversion scheme information includes: Color gamut conversion; or, Chroma format conversion; or, Color compensation; or Color mode conversion.

16. The method according to claim 15, characterized in that The conversion scheme information also includes conversion parameters; The conversion parameters include: compensation matrix; or, a transformation matrix; or, a conversion function; or, Compensation value.

17. A coding method, characterized in that: Determine, according to the acquired first image, base layer data of the first image, enhancement layer data of the first image, and metadata of the first image, wherein the metadata carries conversion information; The base layer data, the enhancement layer data and the metadata are encoded to obtain a code stream.

18. The method according to claim 17, characterized in that The conversion information is conversion indication information, and the conversion indication information includes conversion position information and conversion scheme information, the conversion position information is used to indicate the position where the conversion scheme occurs, and the conversion scheme information is used to indicate the conversion scheme. The conversion occurs before obtaining the second image or after obtaining the second image, and the second image indicates a new image synthesized based on the received code stream.

19. The method according to claim 17 or 18, characterized in that In the case where the conversion occurs before the second image is obtained, the conversion information further includes conversion object information, where the conversion object information is used to indicate a conversion object on which the conversion scheme needs to be executed, and the conversion object includes: Base layer data; or, enhancement layer data; or, The base layer data and the enhancement layer data.

20. The method according to claim 19, characterized in that In the case where the conversion object includes the basic layer data, the conversion scheme information includes: a basic layer data conversion scheme, where the basic layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the basic layer data: Color gamut conversion; Chroma format conversion; Color compensation; or Color mode conversion.

21. The method according to claim 20, characterized in that The basic layer data conversion scheme is also used to indicate conversion parameters used when converting the basic layer data.

22. The method according to claim 21, characterized in that The conversion parameters include: compensation matrix; or, a transformation matrix; or, a conversion function; or, Compensation value.

23. The method according to claim 19, characterized in that In the case where the conversion object includes the enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, where the enhancement layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the enhancement layer data: Color mode conversion; Chroma format conversion; or, Channel conversion, where the channel conversion is used to restore the channels of the enhancement layer data to the original channels.

24. The method according to claim 23, characterized in that The enhancement layer data conversion scheme is also used to indicate conversion parameters used when converting the enhancement layer data.

25. The method according to claim 24, characterized in that The conversion parameters include: a transformation matrix; or, a conversion function; or, Channel conversion function.

26. The method according to any one of claims 18 to 25, characterized in that The conversion information includes channel merging information; the channel merging information is used to indicate: a combination scheme of the channel corresponding to the color mode of the basic layer data and the channel corresponding to the color mode of the enhancement layer data.

27. The method according to claim 26, characterized in that The color mode of at least one of the base layer data and the enhancement layer data is a red, green, blue (RGB) mode or a grayscale mode; Alternatively, the color mode of the base layer data is an RGB mode, and the color mode of the enhancement layer data is any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode; Alternatively, the color mode of the base layer data is any one of a YUV mode, a Lab mode, a HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode, and the color mode of the enhancement layer data is an RGB mode; Alternatively, the color mode of the basic layer data and the color mode of the enhanced layer data are any one of: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode.

28. The method according to claim 27, characterized in that In the case where the color modes of the basic layer data and the enhancement layer data are both RGB modes, the channel merging information is used to indicate: an R channel of the basic layer data is merged with an R channel of the enhancement layer data, a G channel of the basic layer data is merged with a G channel of the enhancement layer data, and a B channel of the basic layer data is merged with a B channel of the enhancement layer data; When the color mode of the base layer data is the RGB mode and the color mode of the enhancement layer data is the grayscale mode, the channel merging information is used to indicate: the R channel of the base layer data is merged with the grayscale value of the enhancement layer data, the G channel of the base layer data is merged with the grayscale value of the enhancement layer data, and the B channel of the base layer data is merged with the grayscale value of the enhancement layer data; When the color mode of the base layer data is a grayscale mode and the color mode of the enhancement layer data is an RGB mode, the channel merging information is used to indicate: an R channel of the enhancement layer data is merged with a grayscale value of the base layer data, a G channel of the enhancement layer data is merged with a grayscale value of the base layer data, and a B channel of the enhancement layer data is merged with a grayscale value of the base layer data; In the case where the color modes of the base layer data and the enhancement layer data are both grayscale modes, the channel merging information is used to indicate that the grayscale value of the base layer data and the grayscale value of the enhancement layer data are merged; In the case where the color mode of the base layer data and the color mode of the enhancement layer data are any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode, the channel merging information is used to indicate that the brightness channel of the base layer data and the brightness channel of the enhancement layer data are merged; In the case where the color modes of the base layer data and the enhancement layer data are both any one of the YUV mode, the Lab mode, the HSV mode, the YCbCr mode, and the ICtCp mode, the channel merging information is used to indicate that the brightness channel of the base layer data and the brightness channel of the enhancement layer data are merged; When the color mode of the base layer data is a grayscale mode and the color mode of the enhancement layer data is any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, and an ICtCp mode, the channel merging information is used to indicate that a channel of the base layer data and a brightness channel of the enhancement layer data are merged; When the color mode of the basic layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, and the color mode of the enhanced layer data is a grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the channel of the enhanced layer data are merged.

29. The method according to claim 20 or 23, characterized in that The color mode conversion includes: Convert the color mode of the basic layer data so that the converted color mode of the basic layer data is the same as the color mode of the enhancement layer data, or convert the color mode of the enhancement layer data so that the converted color mode of the enhancement layer data is the same as the color mode of the basic layer data.

30. The method according to claim 17 or 18, characterized in that In the case where the conversion occurs after the second image is obtained, the conversion scheme information includes: Color gamut conversion; or, Chroma format conversion; or, Color compensation; or Color mode conversion.

31. The method according to claim 30, characterized in that The conversion scheme information also includes conversion parameters; The conversion parameters include: compensation matrix; or, a transformation matrix; or, a conversion function; or, Compensation value.

32. The method according to any one of claims 1 to 31, characterized in that The conversion information includes at least one or more of the following: Color mode information of the base layer data of the first image; Color mode information of the enhancement layer data of the first image; and / or, The color mode information corresponding to the basic layer data of the first image and the enhanced layer data of the first image needs to be transformed and processed.

33. The method according to claim 32, characterized in that When the image color space model of the basic layer data is a first color mode, the color space model when the basic layer data and the enhanced layer data are processed in combination is a second color mode, and the first color mode is different from the second color mode, the transformed and processed color mode information includes a color mode conversion method.

34. The method according to claim 33, characterized in that The first color mode and the second color mode are one of the following: RGB, YUV, YCbCr, Lab, HSV, ICtCp.

35. The method according to any one of claims 1 to 34, characterized in that The conversion information includes at least one or more of the following: Chroma format information of base layer data of the first image; Chroma format information of enhancement layer data of the first image; and / or, The chroma format information corresponding to the base layer data of the first image and the enhancement layer data of the first image that needs to be transformed and processed.

36. The method according to claim 35, characterized in that In the case where the chroma format information of the base layer data of the first image is the first chroma format information, the chroma format information when the base layer data is processed in combination with the enhancement layer data is the second chroma format information, and the first chroma format information is different from the second chroma format information, The chroma format information that needs to be transformed and processed includes chroma format conversion information.

37. The method according to claim 36, characterized in that The first chroma format information and the second chroma format information are one of the following: RGB, grayscale, YUV4:4:4, YUV4:2:2, YUV4:2:

0.

38. The method according to any one of claims 1 to 37, characterized in that: The first image and the second image are HDR or SDR images.

39. A chip, characterized in that: include: Processor and power supply circuit; The power supply circuit is used to supply power to the processor; The processor is used to execute the method of any one of claims 1 to 16 or any one of claims 32 to 38; and / or, the processor is used to execute the method of any one of claims 17 to 38.

40. A decoding device, characterized in that: The decoding device comprises: A processing module, determining target data according to conversion information carried in metadata; wherein the metadata is metadata of a first image, and the target data includes base layer data of the first image and / or enhancement layer data of the first image; or, the target data includes a second image; and the second image is obtained according to the base layer data and the enhancement layer data; A conversion module converts the target data according to the conversion information.

41. A coding device, characterized in that The encoding device comprises: A processing module, which determines, according to the acquired first image, base layer data of the first image, enhancement layer data of the first image, and metadata of the first image, wherein the metadata carries conversion information; The encoding module encodes the base layer data, the enhancement layer data and the metadata to obtain a code stream.

42. A codec, characterized in that: It comprises a memory and a processor, the memory is used to store computer instructions; when the processor executes the computer instructions, it implements any one of claims 1 to 16 or any one of claims 32 to 38; and / or when the processor executes the computer instructions, it implements any one of claims 17 to 38.

43. A coding and decoding system, characterized in that: Includes encoder and decoder; The encoder is used to obtain, according to the first image, base layer data, enhancement layer data and metadata carrying conversion information of the first image, and encoding the base layer data, the enhancement layer data and the metadata to obtain a code stream of the first image, thereby implementing the method according to any one of claims 17 to 38; The decoder is used to determine and convert target data according to conversion information carried by metadata of the first image, thereby implementing the method of any one of claims 1 to 16 or any one of claims 32 to 38.

44. A computer-readable storage medium, characterized in that The storage medium stores a computer program or instruction, which, when executed by a processing device, implements the method of any one of claims 1 to 16 or any one of claims 32 to 38; and / or, when executed by a processing device, implements the method of any one of claims 17 to 38.

45. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed on a processed device, it implements the method of any one of claims 1 to 16 or any one of claims 32 to 38; and / or, when the computer program or instruction is executed on a processed device, it implements the method of any one of claims 17 to 38.

Images