Data processing method, device and system

Abstract

The invention discloses a data processing method, device and system, and relates to the technical field of video or image processing. And the encoder obtains the basic layer data, the enhancement layer data and the metadata of the first image, and encodes the basic layer data, the enhancement layer data and the metadata to obtain a code stream. And the decoder receives and analyzes the code stream to obtain the basic layer data, the enhancement layer data and the metadata of the first image. And the decoder determines and converts target data (namely basic layer data and enhancement layer data or an image obtained by combining the basic layer data and the enhancement layer data) according to conversion information carried by the metadata. Therefore, the decoder determines and converts the target data according to the conversion information carried in the metadata, so that the conversion performed on the target data is matched with the target data, and the decoder converts the target data by adopting the conversion information matched with the target data. Therefore, the dynamic range of the image obtained through conversion can better adapt to the dynamic range supported by the display device, and the display effect is improved.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on October 7, 2023, with application number 202311289650.8 and application name “Encoding Method and Related Devices”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of video or image processing technology, and in particular to a data processing method, device and system. Background Art

[0003] In the field of video or image processing, dynamic range refers to the ratio of the maximum grayscale value (or brightness value) of the pixels of an image to the minimum grayscale value.

[0004] After an electronic device captures a video or image, it can display the video or image on the electronic device or transmit the video or image to other display devices for display on other display devices. Different display devices can display different dynamic ranges of pixels. When displaying a video or image on a display device, the dynamic range of the captured video or image may be different from the dynamic range supported by the display device. Therefore, the dynamic range of the captured video or image needs to be adjusted to the dynamic range supported by the display device. Only in this way can the video or image be successfully displayed on the display device.

[0005] The electronic device can convert the dynamic range of the video or image according to the dynamic range supported by the display device, so that the dynamic range of the converted video or image matches the dynamic range supported by the display device. The electronic device converts the dynamic range of the video or image based on the dynamic range supported by the display device, which affects the coding efficiency. Summary of the invention

[0006] The present application provides a data processing method, device and system for solving the problem that the dynamic range of a video or image is converted by an electronic device based on the dynamic range supported by a display device, which affects the coding efficiency.

[0007] In a first aspect, an embodiment of the present application provides a data processing method. The method is executed by a decoder, and the decoder determines the target data based on the conversion information carried in the metadata. The metadata is metadata of the first image. The target data includes at least one of the basic layer data of the first image and the enhanced layer data of the first image. For example, the target data is the basic layer data of the first image, and the target data is the enhanced layer data of the first image, and the target data is the basic layer data of the first image and the enhanced layer data of the first image. Or the target data includes a second image. The second image is obtained based on the basic layer data of the first image and the enhanced layer data of the first image. The decoder converts the target data based on the conversion information.

[0008] In the present application, the decoder determines and converts the target data according to the conversion information carried in the metadata, so that the conversion of the target data matches the target data. The decoder converts the target data using the conversion information that matches the target data, so that the dynamic range of the converted image can not only better adapt to the dynamic range supported by the display device, but also improve the display effect.

[0009] In a possible implementation, the metadata may be preset or carried in a bitstream. In the case where the metadata is carried in the bitstream, the decoder obtains the metadata from the bitstream. In this way, the bitstream carries the metadata of the first image, so that the decoder can quickly and accurately obtain the metadata, thereby improving conversion efficiency.

[0010] In another possible implementation, the conversion information includes conversion position information and conversion scheme information. The conversion position information is used to indicate the position where the conversion occurs, and the conversion scheme information is used to indicate the conversion scheme used when converting the target data, and the conversion occurs before or after the second image is obtained. In this way, the decoder can determine the conversion scheme to be executed according to the conversion position information, so as to quickly obtain the conversion scheme matching the target data and improve the decoding efficiency.

[0011] In another possible implementation, when the conversion occurs before the second image is obtained, the conversion information further includes conversion object information, the conversion object information is used to indicate the conversion object for which the conversion scheme needs to be executed, and the conversion object includes: basic layer data or enhancement layer data or basic layer data and enhancement layer data. In this way, the decoder determines the conversion scheme matching the conversion object information according to the conversion object information to improve the conversion effect.

[0012] In another possible implementation, when the conversion object includes base layer data, the conversion scheme information includes: a base layer data conversion scheme, and the base layer data conversion scheme is used to indicate that at least one of color gamut conversion, chroma format conversion, color compensation, and color mode conversion is performed on the base layer data. In this way, the decoder performs at least one of color gamut conversion, chroma format conversion, color compensation, and color mode conversion on the base layer data according to the base layer data conversion scheme, so that the converted image can better adapt to the display device and improve the display effect.

[0013] In another possible implementation manner, the base layer data conversion scheme is further used to indicate conversion parameters used when converting the base layer data.

[0014] In another possible implementation, the conversion parameter includes at least one of a compensation matrix, a conversion matrix, a conversion function, and a compensation value. In this way, the decoder can quickly and accurately use the conversion parameters that match the conversion process, and use the conversion parameters that match the conversion process to convert the base layer data, thereby improving the conversion effect.

[0015] In another possible implementation, when the conversion object includes enhanced layer data, the conversion scheme information includes: an enhanced layer data conversion scheme. The enhanced layer data conversion scheme is used to indicate that at least one of color mode conversion, chroma format conversion, and channel conversion is performed on the enhanced layer data. The channel conversion is used to restore the channel of the enhanced layer data to the original channel. In this way, the decoder performs at least one of color mode conversion, chroma format conversion, and channel conversion on the basic layer data according to the basic layer data conversion scheme, so that the converted image can better adapt to the display device and improve the display effect.

[0016] In another possible implementation manner, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the enhancement layer data.

[0017] In another possible implementation manner, the conversion parameter includes: at least one of a conversion matrix, a conversion function, and a channel conversion function.

[0018] In a possible scenario, the conversion information includes channel merging information. The channel merging information is used to indicate a combination scheme of a channel corresponding to a color mode of the base layer data and a channel corresponding to a color mode of the enhancement layer data.

[0019] In another possible implementation, the color modes of the base layer data and the enhancement layer data may include a variety of different combinations, several of which are given below.

[0020] Mode 1: The color mode of at least one of the base layer data and the enhancement layer data is a red, green, blue (RGB) mode or a grayscale mode.

[0021] Mode 2: the color mode of the basic layer data is the RGB mode, and the color mode of the enhanced layer data is any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode.

[0022] Mode 3: the color mode of the basic layer data is any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode; the color mode of the enhanced layer data is RGB mode.

[0023] Mode 4: the color mode of the basic layer data and the color mode of the enhanced layer data are any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode.

[0024] In a possible scenario, according to the different color modes of the enhancement layer data of the basic layer data, the channel merging manner of the basic layer data and the enhancement layer data indicated by the channel merging information is also different. Several possible manners are given below.

[0025] Method 1, when the color modes of the basic layer data and the enhanced layer data are both RGB modes, the channel merging information is used to indicate: the R channel of the basic layer data and the R channel of the enhanced layer data are merged, the G channel of the basic layer data and the G channel of the enhanced layer data are merged, and the B channel of the basic layer data and the B channel of the enhanced layer data are merged.

[0026] Method 2, when the color mode of the basic layer data is the RGB mode and the color mode of the enhanced layer data is the grayscale mode, the channel merging information is used to indicate: the R channel of the basic layer data is merged with the grayscale value of the enhanced layer data, the G channel of the basic layer data is merged with the grayscale value of the enhanced layer data, and the B channel of the basic layer data is merged with the grayscale value of the enhanced layer data.

[0027] Method 3, when the color mode of the basic layer data is the grayscale mode and the color mode of the enhanced layer data is the RGB mode, the channel merging information is used to indicate: the R channel of the enhanced layer data is merged with the grayscale value of the basic layer data, the G channel of the enhanced layer data is merged with the grayscale value of the basic layer data, and the B channel of the enhanced layer data is merged with the grayscale value of the basic layer data.

[0028] Mode 4: When the color modes of the basic layer data and the enhancement layer data are both grayscale modes, the channel merging information is used to indicate that the grayscale value of the basic layer data and the grayscale value of the enhancement layer data are merged.

[0029] Method 5, when the color mode of the basic layer data and the color mode of the enhanced layer data are both any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0030] Method 6, when the color modes of the basic layer data and the enhanced layer data are any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0031] Method 7, when the color mode of the basic layer data is the grayscale mode and the color mode of the enhanced layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, the channel merging information is used to indicate: the channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0032] Method 8, when the color mode of the basic layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, and the color mode of the enhanced layer data is the grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the channel of the enhanced layer data are merged.

[0033] In this way, the decoder merges the channels of the basic layer data and the enhancement layer data using a channel merging scheme that matches the color modes of the basic layer data and the enhancement layer data, thereby improving the display effect of the merged image.

[0034] In another possible implementation, the channel merging method may include but is not limited to: addition, multiplication, etc.

[0035] In another possible implementation, the color mode conversion includes: converting the color mode of the base layer data so that the converted color mode of the base layer data is the same as the color mode of the enhancement layer data. Alternatively, converting the color mode of the enhancement layer data so that the converted color mode of the enhancement layer data is the same as the color mode of the base layer data.

[0036] In another possible implementation, when the conversion occurs after the second image is obtained, the conversion scheme information includes: at least one of: color gamut conversion, chroma format conversion, color compensation, and color mode conversion.

[0037] In another possible implementation, the conversion scheme information further includes conversion parameters. The conversion parameters include: at least one of a compensation matrix, a conversion matrix, a conversion function, and a compensation value.

[0038] In another possible implementation manner, the conversion information includes at least one or more of the following manners 1 to 3.

[0039] Mode 1: color mode information of the basic layer data of the first image.

[0040] Mode 2: color mode information of the enhancement layer data of the first image.

[0041] Mode 3: Color mode information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0042] In another possible implementation, when the image color space model of the basic layer data is a first color mode, the color space model when the basic layer data is processed in combination with the enhancement layer data is a second color mode, and the first color mode is different from the second color mode, the transformed and processed color mode information includes a color mode conversion method. In this way, the decoder converts the image color space model of the basic layer data and / or the image color space model of the enhancement layer data, so that the converted image color space model of the basic layer data or the enhancement layer data matches the color space model when the basic layer data is processed in combination with the enhancement layer data, so that the basic layer data and the enhancement layer data can be effectively merged subsequently to improve the display effect.

[0043] In another possible implementation manner, the first color mode and the second color mode are one of the following: RGB, YUV, YCbCr, Lab, HSV, and ICtCp.

[0044] In another possible implementation manner, the conversion information includes at least one or more of the following manners 1 to 3.

[0045] Mode 1: Chroma format information of the basic layer data of the first image.

[0046] Mode 2: Chroma format information of the enhancement layer data of the first image.

[0047] Mode 3: the chroma format information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0048] In another possible implementation, when the chroma format information of the basic layer data of the first image is the first chroma format information, the chroma format information when the basic layer data is processed in combination with the enhancement layer data is the second chroma format information, and the first chroma format information is different from the second chroma format information, the chroma format information that needs to be transformed and processed includes chroma format conversion information. In this way, the decoder converts the chroma format of the basic layer data and / or the chroma format of the enhancement layer data so that the chroma format of the converted basic layer data or enhancement layer data matches the chroma format when the basic layer data is processed in combination with the enhancement layer data, so that the basic layer data and the enhancement layer data can be effectively merged later to improve the display effect.

[0049] In another possible implementation manner, the first chroma format information and the second chroma format information are one of the following: RGB, grayscale, YUV4:4:4, YUV4:2:2, or YUV4:2:0.

[0050] In another possible implementation manner, the first image and the second image are HDR or SDR images.

[0051] In a second aspect, an embodiment of the present application provides a coding method. The method is performed by an encoder, and the encoder determines the base layer data of the first image, the enhancement layer data of the first image, and the metadata of the first image based on the acquired first image. The metadata carries conversion information. The encoder encodes the base layer data, the enhancement layer data, and the metadata to obtain a code stream.

[0052] Compared to the encoder directly encoding the first image to obtain a bitstream, in the present application, the encoder obtains the base layer data, enhancement layer data and metadata of the first image according to the first image, and encodes the base layer data, enhancement layer data and metadata to obtain a bitstream. In this way, the encoded bitstream contains more information of the first image, which provides a guarantee for decoding to obtain a better quality image.

[0053] In a possible implementation, the conversion information includes conversion position information and conversion scheme information, the conversion position information is used to indicate the position where the conversion occurs, the conversion scheme information is used to indicate the conversion scheme used when converting the target data, and the conversion occurs before or after the second image is obtained. For the relevant contents of the conversion information, the conversion position information, the conversion scheme information, and the conversion scheme, please refer to the relevant contents in the first aspect, which will not be repeated here.

[0054] In a third aspect, an embodiment of the present application provides a method for processing a signal, the method comprising: obtaining first basic layer data, first enhancement layer data and first metadata. The first basic layer data is the basic layer data of a first image, the first metadata is the metadata of the first image, and the first enhancement layer data is the metadata of the first image. According to the conversion information carried in the first metadata, the target data is determined, the target data is the first basic layer data and / or the first enhancement layer data, or the target data is a second image, wherein the second image is an image determined based on the first basic layer data and the first enhancement layer data. According to the conversion information, the target data is converted.

[0055] The first metadata may be preset or carried in the bitstream.

[0056] In a possible implementation, the first image and the second image are HDR or SDR images.

[0057] In a possible implementation, the conversion information is conversion indication information, the conversion indication information includes conversion position information, and when the conversion position is before determining the second image, the conversion indication information also includes conversion object information, and the conversion object information is used to indicate that the target data includes: the first basic layer data, or the first enhancement layer data, or the first basic layer data and the first enhancement layer data.

[0058] In a possible implementation manner, when the target data includes the first basic layer data, the conversion scheme information includes: a basic layer data conversion scheme. The basic layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the first basic layer data:

[0059] Color gamut conversion;

[0060] Color compensation; or

[0061] Color space conversion.

[0062] In a possible implementation manner, the base layer data conversion scheme is further used to indicate conversion parameters used when converting the first base layer data.

[0063] In a possible implementation manner, when the base layer data conversion scheme indicates to perform the color compensation on the first base layer data, the conversion parameter includes any one of the following parameters:

[0064] Color compensation matrix;

[0065] a conversion function; or

[0066] Compensation value and compensation method.

[0067] In a possible implementation, the conversion information is conversion indication information. The conversion indication information includes conversion scheme information. When the target data includes the first enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme. The enhancement layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the first enhancement layer data:

[0068] Color space conversion;

[0069] Channel conversion, where the channel conversion is used to restore the channels of the first enhancement layer data to the original channels.

[0070] In a possible implementation manner, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the first enhancement layer data.

[0071] In a possible implementation manner, when the target data is the second image, the conversion scheme information includes: an image conversion scheme, where the image conversion scheme is used to indicate that at least one of the following conversions is performed on the second image:

[0072] Color gamut conversion; or

[0073] Color compensation.

[0074] In a fourth aspect, an embodiment of the present application provides a coding method, the method comprising: determining first base layer data, first enhancement layer data, and first metadata according to an acquired first image, wherein the first metadata carries conversion information. Encoding the first base layer data, the first enhancement layer data, and the first metadata to obtain a code stream.

[0075] In a possible implementation, the conversion information is conversion indication information, and the conversion indication information includes conversion position information and conversion scheme information, wherein the conversion position information is used to indicate the location where the conversion scheme occurs, and the conversion scheme information is used to indicate the conversion scheme. The first metadata may be preset or carried in the bitstream.

[0076] In a possible implementation manner, the conversion scheme occurs before or after a new HDR image is synthesized based on the received bitstream.

[0077] In a possible implementation manner, when the location where the conversion occurs is before the synthesis of the new HDR image, the conversion indication information further includes conversion object information, where the conversion object information is used to indicate target data for which the conversion scheme needs to be executed, and the target data includes:

[0078] Base layer data; or

[0079] enhancement layer data; or

[0080] The base layer data and the enhancement layer data.

[0081] In a possible implementation manner, when the target data includes the base layer data, the conversion scheme information includes: a base layer data conversion scheme, where the base layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the base layer data:

[0082] Color gamut conversion;

[0083] Color compensation; or

[0084] Color space conversion.

[0085] In a possible implementation manner, the base layer data conversion scheme is also used to indicate conversion parameters used when performing the conversion.

[0086] In a possible implementation manner, when the base layer data conversion scheme indicates to perform the color compensation, the conversion parameter includes any one of the following parameters:

[0087] Color compensation matrix;

[0088] a conversion function; or

[0089] Compensation value and compensation method.

[0090] In a possible implementation manner, when the target data includes the enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, where the enhancement layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the enhancement layer data:

[0091] Color space conversion;

[0092] Channel conversion, where the channel conversion is used to restore the channels of the enhanced layer data to the original channels.

[0093] In a possible implementation manner, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the first enhancement layer data.

[0094] In a possible implementation manner, when the conversion scheme occurs after the new HDR image is synthesized, the conversion scheme information includes: an image conversion scheme, where the image conversion scheme is used to indicate that at least one of the following conversions is performed on the new HDR image:

[0095] Color gamut conversion; or

[0096] Color compensation.

[0097] The embodiments of the present application can preset or transmit the color mode information of the basic layer, and / or the color mode information of the enhancement layer image / enhancement data, and / or the color mode information of the conversion and processing of the basic layer and the enhancement layer image / enhancement data in the metadata, and perform conversion of the basic layer image and / or the enhancement layer image / enhancement data based on this information, so that the conversion is more flexible, and the metadata can achieve multi-format adaptation, which can be conducive to more efficient and reasonable encoding.

[0098] The embodiments of the present application can convert the base layer image and / or the enhancement layer image / enhancement data based on the chromaticity format information of the base layer preset or transmitted in metadata, and / or the chromaticity format information of the enhancement layer image / enhancement data, and / or the chromaticity format information of the conversion and processing of the base layer and the enhancement data, thereby achieving a higher degree of matching with the color gamuts of different display devices, supporting color space conversion, and facilitating more efficient and reasonable encoding.

[0099] The embodiments of the present application can provide information on how to combine the color mode multi-channel of the basic layer with the color mode multi-channel or single channel of the enhancement layer image / enhancement data in preset or transmitted metadata, thereby making the conversion more flexible and enabling flexible adaptation in a variety of ways.

[0100] In a fifth aspect, an embodiment of the present application provides a decoding device, wherein the device includes modules for executing the decoding method in the first aspect or any possible design of the first aspect.

[0101] In a sixth aspect, an embodiment of the present application provides a coding device, wherein the device includes modules for executing the coding method in the second aspect or any possible design of the second aspect.

[0102] In a seventh aspect, an embodiment of the present application provides an electronic device. The electronic device includes at least one processor and a memory, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions as an execution device in the first aspect or any possible implementation of the first aspect, the operation steps of the decoding method in the first aspect or any possible implementation of the first aspect are executed, or when the processor executes the set of computer instructions as an execution device in the second aspect or any possible implementation of the second aspect, the operation steps of the encoding method in any two possible implementations of the first aspect or the second aspect are executed.

[0103] In an eighth aspect, an embodiment of the present application provides a coding and decoding system. The system includes a first electronic device and a second electronic device. The first electronic device is used to execute the decoding method in the first aspect or any possible design of the first aspect, and the second electronic device is used to execute the encoding method in the first aspect or any possible design of the first aspect.

[0104] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, including: computer software instructions; when the computer software instructions are executed in a computing device, the computing device executes the operation steps of the method described in the first aspect or any possible implementation of the first aspect.

[0105] In a tenth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed on a computer, the computing device is caused to execute the operation steps of the method described in the first aspect or any possible implementation of the first aspect.

[0106] The beneficial effects of the third to eighth aspects above can be referred to the description of any implementation in the first or second aspect, and will not be repeated here. Based on the implementation provided by the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 is a schematic diagram of dynamic range mapping;

[0108] Figure 2 is the image of the PQ photoelectric transfer function;

[0109] Figure 3 is the image of HLG photoelectric transfer function;

[0110] Figure 4 is the image of SLF photoelectric transfer function;

[0111] Figure 5 Schematic diagram of the end-to-end process of the HDR object provided by this application;

[0112] Figure 6 Schematic diagram of the end-to-end system architecture of the HDR object provided by this application;

[0113] Figure 7 A schematic diagram of an end-to-end image processing system provided for this application;

[0114] Figure 8 A schematic diagram of a streaming media system workflow provided for this application;

[0115] Fig. 9 A schematic diagram of a streaming media system architecture provided for this application;

[0116] Fig.10 A schematic block diagram of a coding and decoding system used in this application;

[0117] Fig.11 A block diagram of a content providing system provided by the present application;

[0118] Fig.12 A schematic diagram of the workflow of a terminal device provided for this application;

[0119] Fig.13A A schematic diagram of the first image processing process provided in this application;

[0120] Fig. 13B A schematic diagram of the second image processing process provided by this application;

[0121] Fig. 13C A schematic diagram of the third image processing process provided by this application;

[0122] Fig.13D A schematic diagram of the fourth image processing process provided by this application;

[0123] Fig.13E A schematic diagram of the fifth image processing process provided by this application;

[0124] Fig.13F A schematic diagram of the sixth image processing process provided by this application;

[0125] Fig.14 A flowchart of a data processing method provided in this application;

[0126] Fig.15 A schematic diagram of the first conversion process provided for this application;

[0127] Fig.16 A schematic diagram of the second conversion process provided for this application;

[0128] Fig.17 A schematic diagram of the third conversion process provided for this application;

[0129] Fig.18 A schematic diagram of the fourth conversion process provided for this application;

[0130] Fig.19 A schematic diagram of the structure of an encoding device provided in this application;

[0131] Fig. 20 A schematic diagram of the structure of an encoding device provided in this application;

[0132] Fig.21 A schematic diagram of the structure of a decoding device provided by the present application;

[0133] Fig. 22 A schematic diagram of the structure of a decoding device provided in this application. DETAILED DESCRIPTION

[0134] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings.

[0135] The term "at least one" used in this application refers to one, or more than one, including one, two, three and more; "plurality" refers to two, or more than two, including two, three and more.

[0136] The term "at least one" in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where any of a, b, c, ab, ac, bc, or abc can include a single a, a single b, a single c, and can also include multiple a, multiple b, and multiple c.

[0137] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria, and do not indicate the difference in the content, priority or importance of the two criteria.

[0138] In addition, the terms "including" and "having" in the embodiments, claims and drawings of the present application are not exclusive. For example, a process, method, system, product or device including a series of steps or modules / units is not limited to the listed steps or modules, and may also include steps or modules / units that are not listed.

[0139] In order to facilitate understanding of the embodiments of the present application, some concepts or terms involved in the embodiments of the present application are explained.

[0140] Color value: A value corresponding to a specific image primary color component (such as R, G, B or Y). Primary color values ​​can include linear primary color values ​​and nonlinear primary color values.

[0141] Linear primary color value: The linear primary color value is proportional to the light intensity. In one possible case, its value should be normalized to [0,1], referred to as E.

[0142] Non-linear primary color value: Non-linear primary color value is the normalized digital expression value of image information, which is proportional to the digital coding value. In one possible case, its value should be normalized to [0,1], referred to as E′.

[0143] Digital code value: A digital expression of an image signal. Digital code value is used to represent nonlinear primary color values.

[0144] Electro-optical transfer function (EOTF): A conversion relationship from nonlinear primary color values ​​to linear primary color values.

[0145] Metadata: Data carried in a video signal that describes the video source information.

[0146] Dynamic metadata: Metadata associated with each frame of an image that changes from frame to frame.

[0147] Static metadata: Metadata associated with an image sequence that remains constant within the image sequence.

[0148] Luma: represents the combination of nonlinear primary color signals, symbolized by Y'.

[0149] Luminance mapping: The luminance of the source image is mapped to the luminance of the target system.

[0150] Display adaptation: Processing of video signals to adapt the display characteristics of the target display.

[0151] Source picture: The image input in the HDR pre-processing stage.

[0152] Mastering display: A reference display used when editing and producing video signals to determine the effects of video editing and production.

[0153] Linear scene light signal: In HDR video technology, the HDR video signal with content as scene light refers to the scene light captured by the camera / camera sensor, which is generally a relative value; the linear scene light signal is encoded by HLG to obtain the HLG signal, which is a scene light signal and is nonlinear; the scene light signal generally needs to be converted into a display light signal through an Opto-Optical Transfer Function (OOTF) curve before being displayed on a display device.

[0154] Linear display light signal: In HDR video technology, the HDR video signal with content as display light refers to the display light emitted on the display device, generally expressed as an absolute value in nits. The linear display light signal is PQ encoded to obtain a PQ signal, which is a display light signal and a nonlinear signal. The display light signal is generally displayed on the display device according to its absolute brightness.

[0155] Optical-to-optical conversion factor (OOTF): The curve used in video technology to convert one optical signal into another optical signal.

[0156] Luma-chroma-chroma (LCC), luminance and chrominance separate the three components of the video signal.

[0157] Dynamic range: In most fields, dynamic range is used to express the ratio of the maximum value to the minimum value of a variable. In the field of digital imaging, dynamic range expresses the ratio of the maximum grayscale value to the minimum grayscale value of the image pixel. Grayscale value can also be understood as brightness value, with units of nits or candela per square meter (cd / m 2 ), 1nit=1cd / m 2 .

[0158] Generally, the dynamic range of nature is relatively large, and the brightness of the night scene under the starry sky is about 10 -3 cd / m 2 , the sun itself is as bright as 10 9 cd / m 2 , so the dynamic range reaches 10 9 / 10 -3 =10 12 In real scenes in nature, since the brightness of the sun and the brightness of the stars are not obtained at the same time, the dynamic range of the real natural environment scene may not reach 10 12 For real natural environment scenes, the dynamic range is generally 10 -3Up to 10 6 within the range.

[0159] It is understandable that dynamic range is divided into high dynamic range (HDR) or low dynamic range (SDR), and low dynamic range can also be called standard dynamic range. -3 Up to 10 6 An image with a brightness range of 0 to 255 is called a high dynamic range image, and an image with a brightness range of 0 to 255 is called a low dynamic range image. In most current color digital images, the pixels of each channel of R, G, and B are represented by one byte (i.e. 8 bits), that is, the brightness of the pixels of each channel is represented in the range of 0 to 255, where 0 to 255 is the standard dynamic range of the image. The imaging process of a digital camera is actually a mapping of the dynamic range of the real world to the dynamic range supported by the display device, which is often a nonlinear process, such as Figure 1 As shown, Figure 1 is a schematic diagram of dynamic range mapping.

[0160] The optical-electro transfer function (OETF) represents the conversion relationship from linear signals to nonlinear signals of image pixels. The optical-electro transfer function can include the following three types: Perceptual Quantizer (PQ) optical-electro transfer function, Hybrid Log-Gamma (HLG) optical-electro transfer function and Scene Luminance Fidelity (SLF) optical-electro transfer function. These three optical-electro transfer functions are optical-electro transfer functions specified by the Audio Video Coding Standard (AVS).

[0161] (1) PQ photoelectric transfer function

[0162] The PQ photoelectric transfer function is a perceptual quantization photoelectric transfer function proposed based on the brightness perception model of the human eye. Figure 2 is an image of a PQ photoelectric transfer function. The computing device may use the PQ photoelectric transfer function to represent the conversion relationship between the linear signal value of the image pixel and the nonlinear signal value in the PQ domain. The PQ photoelectric transfer function may be represented by formula (1).

[0163]

[0164] Wherein, L represents the linear signal value, and its value is normalized to [0, 1]. L' represents the nonlinear signal value, and its value range is [0, 1]. m1,

[0165] m2, c1, c2, c3 are PQ photoelectric transfer coefficients, and m1, m2, c1, c2, c3 satisfy the following relationship:

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] Taking the image color model as the three primary color light mode (red, green, blue, RGB) as an example, the conversion process of the RGB image using the PQ photoelectric transfer function can be expressed by formula (2).

[0172]

[0173] (2) HLG Photoelectric Transfer Function

[0174] The HLG photoelectric transfer function is improved based on the traditional Gamma curve. Figure 3 is the image of HLG photoelectric transfer function, such as Figure 3 As shown, the HLG photoelectric transfer function applies the traditional Gamma curve in the low segment and supplements the log curve in the high segment. The HLG photoelectric transfer function represents the conversion relationship from the linear signal value of the image pixel to the nonlinear signal value of the HLG domain. The HLG photoelectric transfer function can be expressed as formula (3).

[0175]

[0176] Wherein, L represents the linear signal value, and its value range is [0, 12], L' represents the nonlinear signal value, and its value range is [0, 1], a, b, c are HLG photoelectric transfer coefficients, a = 0.17883277, b = 0.28466892, c = 0.55991073.

[0177] (3) SLF Photoelectric Transfer Function

[0178] The SLF photoelectric transfer function is the optimal curve obtained based on the brightness distribution of the HDR scene, under the premise of satisfying the optical characteristics of the human eye. Figure 4 As shown, Figure 4is the image of the SLF photoelectric transfer function. The SLF photoelectric transfer curve represents the conversion relationship between the linear signal value of the image pixel and the nonlinear signal value in the SLF domain. The conversion relationship between the linear signal value of the image pixel and the nonlinear signal value in the SLF domain can be expressed as formula (4).

[0179]

[0180] The SLF photoelectric transfer function can be expressed as formula (5).

[0181]

[0182] Wherein, L represents the linear signal value, and its value is normalized to [0,1]; L' represents the nonlinear signal value, and its value range is [0,1]; p, m, a, and b are SLF photoelectric transfer coefficients, p = 2.3, m = 0.14, a = 1.12762, and b = -0.12762.

[0183] Linear space: In this application, linear space refers to the space where linear optical signals are located.

[0184] Nonlinear space: In this application, nonlinear space refers to the space where linear light signals are converted using nonlinear curves; nonlinear curves commonly used in HDR include PQ EOTF-1 curve, HLG OETF curve, etc., and nonlinear curves commonly used in SDR include gamma curves; it is generally believed that linear light signals are visually linear relative to the human eye after being encoded by the above nonlinear curves. It should be understood that nonlinear space can be considered as visual linear space.

[0185] Gamma correction: Gamma correction is a method of nonlinear color editing of images. It can detect the dark and light parts of the image signal and increase the ratio of the two, thereby improving the image contrast effect. The photoelectric conversion characteristics of current display screens, photographic films and many electronic cameras can be nonlinear. The relationship between the output and input of these nonlinear components can be expressed by a power function, that is, output = (input) γ.

[0186] The nonlinear conversion of the color values ​​output by the device is due to the fact that the human visual system is not linear, and humans perceive visual stimuli through comparison. The external world strengthens the stimulus in a certain proportion, and for humans, this stimulus increases evenly. Therefore, for human perception, the physical quantity that increases in geometric progression is uniform. In order to display the input color according to the laws of human vision, it is necessary to convert the linear color value into a nonlinear color value through the nonlinear conversion in the form of the above power function. The value of gamma γ can be determined according to the photoelectric conversion curve of the color space.

[0187] Color space: Color can be the different perceptions of the eyes for light of different frequencies, or it can represent objectively existing light of different frequencies. Color space is a color range defined by a coordinate system established by people to represent colors. The color gamut mode and color model together define a color space. Among them, the color model is an abstract mathematical model that represents color with a set of color components. The color model can include, for example, a three-primary color light mode (red green blue, RGB) and a four-color printing mode (cyan magenta yellow key plate, CMYK). The color gamut refers to the sum of colors that a system can produce. Exemplarily, Adobe RGB and sRGB are two different color spaces based on the RGB model. In this application, the color model can also be referred to as a color mode.

[0188] RGB space refers to a space that uses the brightness of the three primary colors of red, green, and blue to quantitatively represent video or image signals. YCC space is a color space that represents the separation of brightness and chromaticity. The three components of a YCC space video or image signal represent brightness, chromaticity, and chromaticity, respectively. Common YCC space video signals include YUV, YCbCr, and ICtCp. In this application, brightness and chromaticity separation can be referred to as brightness and color separation, and brightness and chromaticity not separated can be referred to as brightness and color not separated.

[0189] Each device, such as a monitor or printer, has its own color space and can only produce colors within its gamut. When you move an image from one device to another, the colors of the image may change on different devices because each device converts and displays RGB or CMYK according to its own color space.

[0190] In order to obtain images with a higher dynamic range, the bit width of the image is usually greater than or equal to 10 bits (bit, b). Common coding standards that support HDR objects include H.266, H.265 and high efficiency image format (HEIF). The Joint Photographic Experts Group (JPEG) and H.264 only support 8-bit coding and cannot support HDR videos and HDR images well.

[0191] During the distribution or transmission process of HDR images or HDR videos, it may be necessary to convert the dynamic range of HDR images or HDR videos to the dynamic range supported by the display device (such as a TV or iPad). The dynamic range of the converted HDR images or HDR videos is adapted to the display device so that the HDR images or HDR videos can be displayed on the display device. The following two solutions can usually be used to convert the dynamic range of HDR images or HDR videos.

[0192] (1) Single-layer transcoding solution.

[0193] HDR images or HDR videos may need to be transcoded during distribution or transmission. The current HDR transcoding solution is to directly transcode HDR images or HDR videos. After decoding, HDR images or HDR videos obtain HDR pixel values ​​and HDR format identifiers and information. HDR pixel values ​​usually need to be processed (such as image up / down sampling, enhancement, etc.). The HDR format identifier and information, as well as the processed HDR pixel values ​​are then sent to the encoder for re-encoding. The re-encoded HDR images or HDR videos are distributed to other devices.

[0194] This solution of directly transcoding HDR images or HDR videos may cause anomalies in the transcoded HDR images or HDR videos, affecting the user experience. Because when a single-layer encoded HDR bitstream is transcoded (decoded and then re-encoded), it depends on the transcoding system to correctly send all the HDR information obtained by the decoder to the encoder. If the information is lost, the transcoding system will not be able to re-encode in the correct format, resulting in an erroneous bitstream, causing abnormal visual effects when users watch the video, seriously affecting the user experience. Currently, many distribution and transcoding systems on the market have not been upgraded to support the correct transmission of HDR information from the decoder to the encoder, and mainstream mobile phone and application manufacturers have received many user complaints in this regard.

[0195] (2) Two-layer image conversion scheme.

[0196] This scheme realizes the conversion of the dynamic range of HDR images or HDR videos by converting the color gamut range of HDR images or HDR videos. The encoder transmits the base layer data and the enhancement layer data to the decoder, and the decoder receives and merges the base layer data and the enhancement layer data to obtain an image with a new dynamic range supported by the display device. The color gamut range of the base layer data transmitted by the encoder should match the color gamut range of the display device. If the color gamut range of the encoder base layer data does not match the color gamut range of the display device, the encoder should convert the color gamut range of the base layer data to obtain the converted base layer data that matches the color gamut range of the display device, and transmit the converted base layer data to the display device. For example, the color gamut range of the base layer data is P3, and the color gamut range of the display device is Rec.2020. In this case, the encoder converts the color gamut range of the base layer data from P3 to Rec.2020 to obtain the converted base layer data. The encoder sends the converted base layer data and the enhancement layer data to the decoder. Exemplarily, the encoder can use a color gamut mapping matrix to convert the color gamut range of the base layer data from P3 to Rec.2020.

[0197] This solution of adjusting the color gamut range of HDR images or HDR videos requires re-encoding and transmission adaptation if the color gamut of the display device changes, which is not flexible enough. In addition, only the color gamut range is adjusted during the conversion process, resulting in low encoding efficiency.

[0198] Based on this, the present application provides a method for data processing. The method includes: an encoder obtains a first image, and the encoder determines the basic layer data, enhancement layer data and metadata of the first image based on the first image. The encoder encodes the basic layer data, enhancement layer data and metadata to obtain a code stream. The decoder receives and parses the code stream to obtain the basic layer data, enhancement layer data and metadata of the first image. And the decoder determines and converts the target data (i.e., the basic layer data or the enhancement layer data or the image obtained by combining the basic layer data and the enhancement layer data) according to the conversion information carried in the metadata. In this way, the decoder determines and converts the target data according to the conversion information carried in the metadata, so that the conversion of the target data matches the target data, and the decoder converts the target data using the conversion information matching the target data, so that the dynamic range of the converted image can not only be more adaptable to the dynamic range supported by the display device, but also improve the display effect.

[0199] The encoder may also be called the encoding side, encoding device or encoding end. The encoder encodes the acquired image to obtain a bit stream.

[0200] The decoder, which may also be called the decoding side, decoding device or decoding end, decodes the received bit stream to obtain an image.

[0201] Base layer data may also be referred to as a base layer image, a base layer, a base data, a base image, a base image, a base data, a base layer, a base layer data or a base image. The base layer data may be an SDR image or an HDR image with a lower dynamic range.

[0202] Enhanced layer data may also be referred to as enhanced layer image (gain map), enhanced layer, enhanced image, gain layer, gain layer data, gain layer image or gain image. The enhanced layer data may include some image detail information. In this way, the image detail information contained in the enhanced layer data is used to supplement the basic layer data, thereby synthesizing an HDR image with better display effect.

[0203] For ease of description, the terms "SDR object" and "HDR object" are used in some embodiments of the present application. An SDR object can be a static SDR image (also referred to as an SDR image, an SDR photo, an SDR picture, etc.), an SDR video or other types of dynamic SDR images, or a frame of an SDR video or a dynamic SDR image. An HDR object can be a static HDR image (also referred to as an HDR image, an HDR photo, an HDR picture, etc.), an HDR video or other types of dynamic HDR images, or a frame of an HDR video or a dynamic HDR image.

[0204] It is understandable that, for the sake of ease of description, some embodiments of the present application take HDR images as an example to introduce the technical solution of the present application. However, it is understandable that these embodiments can be applied not only to HDR images, but also to other HDR objects, such as HDR videos, dynamic HDR images, and a frame of an HDR video or dynamic HDR image.

[0205] HDR objects are an end-to-end process if you want to get a better experience. Figure 5 The end-to-end process diagram of the HDR object provided for this application is as follows Figure 5 As shown, the original video file (also called a master file) is obtained through material production (such as shooting video, making computer graphics (CG) video, etc.), editing, color grading and other procedures. Then, based on the dynamic metadata of the original video file. After encoding the original video file and the dynamic metadata, a compressed video is obtained. The compressed video is distributed / transmitted (the distribution end is such as a streaming media server, CDN, etc., and the distribution end often needs to be transcoded, that is, the process of decoding first and then encoding) to the terminal device (for example, a computer, set-top box, mobile phone, tablet computer, etc.). The terminal device decodes the compressed video to obtain the decompressed video, and then displays it to the user through a display device (for example, a monitor, TV, etc.).

[0206] Figure 5 The end-to-end process for the described HDR object can be used Figure 6 The system shown is complete, Figure 6 The end-to-end system architecture diagram of the HDR object provided by this application is as follows Figure 6 As shown, the system includes: front-end equipment, transmission link, and terminal display equipment.

[0207] Among them, the front-end device is used to capture or produce HDR content or SDR content (for example, HDR video or image, SDR video or image).

[0208] In a possible embodiment, the front-end device can also be used to extract corresponding metadata from the HDR content. The metadata may include conversion information and dynamic metadata and static metadata corresponding to the HDR content. The front-end device can send the HDR content and metadata to the terminal display device through a transmission link. Specifically, the HDR content and metadata can be transmitted in the form of one data packet, or respectively in two data packets, which is not specifically limited in the embodiments of the present application.

[0209] In a possible embodiment, the terminal display device can be used to receive metadata and HDR content, and extract the conversion information contained in the corresponding metadata and the information of the terminal display device according to the HDR content, obtain the conversion position information and the conversion scheme information to convert the HDR content, convert it into display content adapted to the HDR display device or SDR display device in the terminal display device, and display it. It should be understood that in different embodiments, the terminal display device may include a display device with a display capability of a lower dynamic range or a higher dynamic range than the HDR content generated by the front-end device, and this application is not limited thereto.

[0210] In a possible embodiment, the front-end device and the terminal display device in the present application may be independent and different physical devices. For example, the front-end device may be a video acquisition device or a video production device, wherein the video acquisition device may be a camera, a camera, an image plotter, etc. The terminal display device may be a device with a video playback function such as virtual reality (VR) glasses, a mobile phone, a tablet, a television, a projector, etc.

[0211] In a possible embodiment, the transmission link between the front-end device and the terminal display device may be a wireless connection or a wired connection, wherein the wireless connection may adopt technologies including, for example, long term evolution (LTE), fifth generation (5G) mobile communications, and future mobile communications. The wireless connection may also include technologies such as wireless-fidelity (WiFi), Bluetooth, and near field communication (NFC). The wired connection may include an Ethernet connection, a local area network connection, and the like. This is not specifically limited.

[0212] The present application may also integrate the functions of the front-end device and the functions of the terminal display device on the same physical device, for example, a terminal device such as a mobile phone or a tablet with a video shooting function. The present application may also integrate some functions of the front-end device and some functions of the terminal display device on the same physical device. There is no specific limitation on this.

[0213] Figure 7 A schematic diagram of an end-to-end image processing system provided for this application, in Figure 7 In the example, HDR content or SDR content takes HDR video as an example. The image processing system includes: an HDR preprocessing module, an HDR video encoding module, an HDR video decoding module, and a conversion module.

[0214] Among them, the HDR preprocessing module and the HDR video encoding module can be located in Figure 6 The front-end device shown in the figure, the HDR video decoding module and the conversion module can be located in Figure 6 The terminal display device shown.

[0215] HDR pre-processing module: used to extract dynamic metadata (e.g., color mode, chroma format, channel information, conversion position, conversion scheme, etc.) from HDR video, and process the HDR video according to the dynamic metadata and the HDR video for display. Figure 7 The image processing system determines the conversion scheme based on the display capability of the display device of the image processed. The HDR preprocessing module obtains HDR metadata based on the dynamic metadata determined by the conversion scheme and transmits it. The HDR video can be captured or processed by a colorist; the display capability of the target display device is the brightness range that the target display device can display.

[0216] HDR video encoding module: used to encode HDR video and HDR metadata according to the video compression standard (for example, AVS or HEVC standard) (for example, embedding HDR metadata into the user-defined part of the bitstream) and output the corresponding bitstream (AVS or HEVC bitstream).

[0217] HDR video decoding module: used to decode the generated bitstream (such as AVS bitstream, AVC / H.264 bitstream, HEVC / H.265 bitstream, VVC / H.266 bitstream, etc.) according to the standard corresponding to the bitstream format, and output the decoded HDR video and HDR metadata.

[0218] Conversion module: used to determine the conversion scheme according to the conversion information in the decoded HDR metadata, and convert the decoded HDR video (i.e. HDR adaptation processing or SDR adaptation processing), and send the converted HDR adaptation video to the HDR display terminal for display, or send the SDR adaptation video to the SDR display terminal for display.

[0219] Exemplarily, the HDR pre-processing module may exist in a video acquisition device or a video production device.

[0220] Exemplarily, the HDR video encoding module may exist in a video acquisition device or a video production device.

[0221] Exemplarily, the HDR video decoding module may exist in a set-top box, a television display device, a mobile terminal display device, and a video conversion device for online live broadcasting, online video applications, and the like.

[0222] Exemplarily, the conversion module may exist in a set-top box, a television display device, a mobile terminal display device, and a video conversion device for a network live broadcast, a network video application, etc. More specifically, the conversion module may exist in the form of a chip or a software program in a set-top box, a television display, a mobile terminal display, and may exist in the form of a software program in a video conversion device for a network live broadcast, a network video application, etc.

[0223] In a possible embodiment, when the conversion module and the HDR video decoding module are both present in a set-top box, the set-top box can complete the functions of receiving, decoding and converting the video code stream, and the set-top box sends the decoded video data to a display device through a high definition multimedia interface (HDMI) for display, so that the user can enjoy the video content.

[0224] Above Figures 5 to 7 The end-to-end process described can be completed using a streaming media system. Figure 8 A schematic diagram of a streaming media system workflow provided for this application, such as Figure 8As shown, the streaming media system includes: a content creation module, a video encoding module and a video stream transmission module. The content creation module generates the required content data, such as video or audio. The video encoding module encodes the content data generated by the content creation module through an encoder. The video stream transmission module transmits the encoded video in the form of a code stream.

[0225] In some possible situations, the streaming media system further includes a video stream transcoding module. The video stream transcoding module can convert the format of the video stream into a code stream format of a common transmission protocol of an OTT (over-the-top) device, for example, the protocol includes but is not limited to Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG HTTP Dynamic Adaptive Streaming overHTTP (MPEG-DASH), Real-time Transport protocol (RTP), Real Time Messaging Protocol (RTMP) or any combination thereof.

[0226] In some possible situations, the streaming media system further includes a video stream storage module, which stores the original format of the video stream and / or the converted multiple code stream formats for easy use.

[0227] In some possible situations, the streaming media system also includes a video stream encapsulation module. The video stream encapsulation module encapsulates the video stream to generate an encapsulated video stream, which can be called a video streaming media package. Exemplarily, the video streaming media package can be generated according to the transcoded video stream or the stored video stream.

[0228] In some possible scenarios, the streaming media system also includes a content distribution network (CDN). The CDN is used to distribute video streaming media packages to multiple OTT devices, such as mobile phones, computers, tablets, home projectors, etc.

[0229] It should be noted that video encoding, video stream transmission, video stream transcoding, video stream storage, video streaming package generation and CDN can all be achieved using the cloud platform. Fig. 9 A schematic diagram of a streaming media system architecture provided for this application, such as Fig. 9As shown, the streaming media system includes: a client device (or client for short), a CDN, and a cloud platform.

[0230] The user on the client sends a play or playback request to the cloud platform. In some possible examples, the content of the request may be the title of the movie or TV program being played.

[0231] The cloud platform makes a decision, replies to the client, and sends the address of the content requested by the user on the CDN to the client. In some possible examples, the content sent to the client can be a URL (uniform resource locator) link. Specifically, the playback application service in the cloud platform checks user authorization and permission, and then considers individual client characteristics and current network conditions to determine which specific files are needed to process the playback request. It should be noted that the content delivery network (CDN) regularly reports the operating status, learned routes, and available content (files) to the cache control service in the cloud platform.

[0232] The client then requests the CDN to play the content based on the address, and the CDN provides the content to the client, ultimately completing the user's request.

[0233] The above text describes the process of front-end device acquisition, transmission link or CDN distribution / transmission, and display device display of HDR objects or SDR objects in combination with the accompanying drawings. The following text describes the encoding and decoding system used in this application in combination with the accompanying drawings. Fig.10 A schematic block diagram of a coding system used in the present application, for example, the video coding system 1000 (or simply referred to as the coding system 1000) of the present application can be used. The video encoder 1015 (or simply referred to as the encoder 1015) and the video decoder 1022 (or simply referred to as the decoder 1022) in the video coding system 1000 represent devices that can be used to perform various techniques according to various examples described in the present application.

[0234] like Fig.10 As shown, the coding and decoding system 1000 includes a source device 1010, and the source device 1010 is used to provide coded data such as coded images to a target device 1020 for decoding the coded data.

[0235] The source device 1010 includes an encoder 1015 , and optionally, may further include an image source 1011 , a preprocessor 1013 (or a preprocessing unit, an image preprocessor, etc.), a communication interface 1017 or a communication unit 1017 .

[0236] The image source 1011 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0237] In order to distinguish the processing performed by the pre-processor 1013 or the pre-processing unit 1013 , the image or image data 1012 may also be referred to as a raw image or raw image data 1012 .

[0238] The preprocessor 1013 is used to receive (original) image data 1012 and preprocess the image data 1012 to obtain a preprocessed image 1014 or preprocessed image data 1014. For example, the preprocessing performed by the preprocessor 1013 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment, or denoising. It is understood that the preprocessing unit 1013 may be an optional component.

[0239] The encoder 1015 is configured to receive the pre-processed image data 1014 and provide encoded image data 1016 .

[0240] The communication interface 1017 in the source device 1010 can be used to receive the encoded image data 1016 and send the encoded image data 1016 (or any other processed version) to another device such as the target device 1020 or any other device through the communication channel 1030 for storage or direct reconstruction.

[0241] The target device 1020 includes a decoder 1022 (eg, a video decoder 1022 (or simply decoder 1022 )), and may additionally or optionally include a communication interface 1021 or a communication unit 1021 , a post-processor 1024 (or a post-processing unit 1024 ), and a display device 1026 .

[0242] The communication interface 1021 in the target device 1020 is used to receive the encoded image data 1016 (or any other processed version) directly from the source device 1010 or from any other source device such as a storage device, for example, the storage device is a coded image data storage device, and provide the encoded image data 1016 to the decoder 1022.

[0243] The communication interface 1017 and the communication interface 1021 can be used to send or receive the encoded image data 1016 or the encoded data through a direct communication link between the source device 1010 and the target device 1020, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0244] For example, the communication interface 1017 may be used to encapsulate the encoded image data 1016 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0245] The communication interface 1021 corresponds to the communication interface 1017 , for example, and can be used to receive transmission data, and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 1016 .

[0246] Both the communication interface 1017 and the communication interface 1021 can be configured as follows Fig.10 A unidirectional communication interface, or a bidirectional communication interface, is indicated by an arrow pointing from the source device 1010 to the corresponding communication channel 1030 of the target device 1020, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0247] The decoder 1022 is configured to receive the encoded image data 1016 and provide decoded image data 1023 or a decoded image 1023 .

[0248] The post-processor 1024 in the target device 1020 is used to post-process the decoded image data 1023 (also referred to as reconstructed image data) such as the decoded image 1023 to obtain the post-processed image data 1025 such as the post-processed image 1025. The post-processing performed by the post-processing unit 1024 may include, for example, color format conversion (e.g., conversion from YCbCr to RGB), color matching, cropping or resampling, or any other processing for generating the decoded image data 1023 for display by the display device 1026, etc.

[0249] The display device 1026 in the target device 1020 is used to receive the post-processed image data 1025 to display the image to a user or viewer, etc. The display device 1026 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0250] although Fig.10 The source device 1010 and the target device 1020 are shown as independent devices, but the device embodiment may also include the source device 1010 and the target device 1020 or the functions of the source device 1010 and the target device 1020 at the same time, that is, the source device 1010 or the corresponding function and the target device 1020 or the corresponding function at the same time. In these embodiments, the source device 1010 or the corresponding function and the target device 1020 or the corresponding function may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0251] According to the description, Fig.10 The presence and (exact) division of different units or functions in the source device 1010 and / or the target device 1020 shown may vary depending on the actual device and application, which will be obvious to the skilled person.

[0252] Combine the following Fig.11 describe Fig.10 The content providing system used by the encoding and decoding system, Fig.11 1 is a block diagram of a content providing system provided by the present application. The content providing system 1100 includes a capture device 1110, a terminal device 1120 and an (optional) display 1130. The capture device 1110 communicates with the terminal device 1120 via a communication link 1140. The communication link may include Fig.10 The communication channel 1030 and the communication link 1140 include but are not limited to WiFi, Ethernet, wired, wireless (3G / 4G / 5G), USB or any combination thereof.

[0253] Capture device 1110 can implement Figure 6The function of the front-end device in the video capture device 1110. The capture device 1110 generates data and can encode the data using the encoding method shown in the above embodiment. Alternatively, the capture device 1110 can distribute the data to a streaming media server (not shown in the figure), and the server encodes the data and transmits the encoded data to the terminal device 1120. The capture device 1110 includes but is not limited to a camera, a smart phone or a tablet computer, a computer or a laptop computer, a video conferencing system, a personal digital assistant (PDA), a vehicle-mounted device or any combination thereof. For example, the capture device 1110 may include the above-mentioned source device 1010. When the data includes video, the encoder 1015 in the capture device 1110 can actually perform video encoding processing. When the data includes audio (i.e., sound), the encoder 1015 in the capture device 1110 can actually perform audio encoding processing. In some actual scenarios, the capture device 1110 distributes the encoded video data and the encoded audio data by multiplexing the encoded video data and the encoded audio data together. In other actual scenarios, such as in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. The capture device 1110 distributes the encoded audio data and the encoded video data to the terminal device 1120 respectively.

[0254] The terminal device 1120 in the content providing system 1100 receives and regenerates the encoded data. The terminal device 1120 can be a device with data receiving and recovery capabilities, such as a smart phone or tablet computer 1121, a computer or laptop computer 1122, a network video recorder (NVR) / digital video recorder (DVR) 1123, a television 1124, a set top box (STB) 1125, a video conferencing system 1126, a video surveillance system 1127, a PDA 1128, a vehicle-mounted device 1129 or any combination thereof, or such a device capable of decoding the above-mentioned encoded data. For example, the terminal device 1120 may include the above-mentioned target device 1020. When the encoded data includes video, the decoder 1022 in the terminal device preferentially performs video decoding. When the encoded data includes audio, the decoder 1022 in the terminal device preferentially performs audio decoding processing. The terminal device 1120 can be a video playback application, a streaming media playback application, a streaming media playback platform, or a live broadcast platform, etc. running on the terminal device.

[0255] For terminal devices with a display, such as a smartphone or tablet 1121, a computer or laptop 1122, an NVR / DVR 1123, a TV 1124, a PDA 1128, or a vehicle-mounted device 1129, the terminal device can send the decoded data to its display. For terminal devices without a display, such as an STB 1125, a video conferencing system 1126, or a video surveillance system 1127, the device is connected to an external display 1130 to receive and display the decoded data.

[0256] When each device in the system performs encoding or decoding, the image encoding device or image decoding device shown in the above embodiments may be used. Fig.12 for Fig.11 Schematic diagram of an example structure of terminal device 1120. Fig.12 A schematic diagram of a work flow of a terminal device provided by this application is provided in this application, such as Fig.12 As shown, after the terminal device 1120 receives the bit stream from the capture device 1110, the protocol processing unit 1210 analyzes the transmission protocol of the bit stream. The transmission protocol includes but is not limited to: RTSP, HTTP, HLS, MPEG-DASH, RTP, RTMP or any combination thereof.

[0257] After the protocol processing unit 1210 processes the bit stream, a stream file is generated. The stream file is output to the demultiplexing unit 1220. The demultiplexing unit 1220 can separate the multiplexed data into coded audio data and coded video data. As described above, in other practical scenarios, such as in a video conferencing system, the coded audio data and the coded video data are not multiplexed. In this case, the coded data is not transmitted to the video decoder 1230 and the audio decoder 1240 through the demultiplexing unit 1220.

[0258] The demultiplexing unit 1220 processes the stream file to generate a video elementary stream (ES), an audio ES, and optional subtitles. The video decoder 1230, including the decoder 1022 described in the above embodiment, decodes the video ES to generate a video frame by the decoding method shown in the above embodiment, and sends the video frame to the synchronization unit 1260. The audio decoder 1240 decodes the audio ES to generate an audio frame, and sends the audio frame to the synchronization unit 1260. Alternatively, the video frame can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 1260. Similarly, the audio frame can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 1260.

[0259] The synchronization unit 1260 synchronizes the video frames and the audio frames and provides the video / audio to the video / audio display 1270. For example, the synchronization unit 1260 synchronizes the presentation of the video and audio information. The information may be encoded in the syntax using the timestamps associated with the encoded audio and visual data representations and the timestamps associated with the data stream transmission.

[0260] If the bitstream includes subtitles, the subtitle decoder 1250 decodes the subtitles, synchronizes them with the video frames and the audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 1280 .

[0261] The present application is not limited to the above-mentioned system, and the image encoding device or image decoding device in the above-mentioned embodiments can be used in other systems such as automobiles.

[0262] The above text describes the coding and decoding system that the present application should have in conjunction with the accompanying drawings. The following text describes the workflow diagram of the image processing provided by the present application in conjunction with the accompanying drawings. The image processing process can be performed by an electronic device, and the electronic device includes a decoding module and a processing module. In some possible examples, the electronic device may also include a display module for displaying the obtained target image.

[0263] The decoding module may include any decoder (HEVC, JPEG, ProRes, HEIF). The decoding module uses the decoder to decode the received bitstream to obtain base layer data, enhancement layer data and metadata. The base layer data and enhancement layer data obtained in this process include image data in any color space form such as RGB or YUV. It should be noted that this application does not limit the bitstream format. In terms of color space, it can be YUV or RGB. In terms of the bit width of the data, the bitstream can be 8 bits, 10 bits, or 12 bits, etc. In some embodiments, the decoding module can obtain source data from the SEI of HEVC or VVC, the user-defined NAL unit or the reserved package unit, or it can obtain the metadata from the JFIF-encapsulated APP extension information or the MP4-encapsulated data segment, etc.; it can also obtain the metadata from other file locations, such as the location after the EOI (end of image) of the complete JPEG file.

[0264] It should be noted that metadata mainly includes source data format or image features and conversion information, etc., and one or more metadata information units. The metadata information unit includes coordinate information, image features and conversion information, etc.

[0265] In some embodiments, the decoding module can obtain enhancement layer data from the SEI of HEVC or VVC, a user-defined NAL unit or a reserved packet unit, or from the APP extension information encapsulated in JFIF or the data segment encapsulated in MP4, etc.; the enhancement layer data can also be obtained from other file locations indicated in the metadata, such as the position after the EOI of the complete JPEG file.

[0266] The enhancement layer data may be: one pixel value corresponds to one enhancement layer data, or multiple pixel values ​​correspond to one enhancement layer data, wherein the enhancement layer data may contain multiple values, and the number of values ​​contained in the enhancement layer data corresponding to each pixel is the same.

[0267] The electronic device can use a decoding module or a processing module to perform different processing on the basic layer data and the enhanced layer data to obtain a target image, which is described below in different situations.

[0268] Scenario 1: Fig.13A The first image processing flow chart provided in this application is as follows: Fig.13A As shown, the decoding module uses the decoder to obtain the base layer data, the enhancement layer data and the metadata from the bit stream, and the processing module uses the metadata to process the base layer data to obtain the processed base layer data, and merges the processed base layer data and the enhancement layer data to obtain the target image. The processing module transmits the target image to the display module for display.

[0269] Scenario 2: Fig. 13B The schematic diagram of the second image processing process provided by this application is as follows: Fig. 13B As shown, the decoding module uses the decoder to obtain the base layer data, the enhancement layer data and the metadata from the bit stream. The decoding module synthesizes the base layer data and the enhancement layer data to obtain an HDR image (or HDR image 1), and transmits the HDR image 1 to the processing module. The processing module performs processing 1 on the HDR image 1 to obtain an HDR image 2. The processing module performs processing 2 on the HDR image 2 using the metadata to obtain an HDR image 3. The processing module transmits the HDR image 3 to the display module for display.

[0270] The dynamic range of the base layer data is smaller than the dynamic range of the HDR image (ie, HDR image 1) determined based on the base layer data and the enhancement layer data.

[0271] For example, in some embodiments, the base layer data may be an SDR image, and the HDR image 1 is an HDR image. The HDR image and the SDR image have at least one of different photoelectric conversion functions, or different color gamuts, or different dynamic ranges.

[0272] For another example, in some other embodiments, the base layer data may be an HDR image, but the dynamic range of the HDR image is smaller than the dynamic range of the HDR image 1 .

[0273] For example, assuming that the dynamic range of the base layer data is DRx1 to DRx2, and the dynamic range of HDR image 1 is DRy1 to DRy2. In some embodiments, DRy1 is less than DRx1, and DRy2 is greater than DRx2. In other embodiments, DRy1 is less than DRx1, and DRy2 is equal to DRx2. In other embodiments, DRy1 is equal to DRx1, and DRy2 is greater than DRx2. In other words, the base layer data can be considered to be an image with a dynamic range smaller than that of HDR image 1.

[0274] In some embodiments, the resolution of the base layer data may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0275] In some embodiments, the resolution of the enhancement layer data may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0276] In some embodiments, the resolution of the HDR image 1 may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0277] In some embodiments, the resolution of the HDR image 2 may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0278] In some embodiments, the resolution of the HDR image 3 may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0279] In some embodiments, the resolution of the base layer data may be the same as the resolution of the enhancement layer data. For example, the resolutions of the base layer data and the enhancement layer data are both 4K.

[0280] In other embodiments, the resolution of the base layer data may be different from the resolution of the enhancement layer data. For example, in some embodiments, the resolution of the base layer data may be greater than the resolution of the enhancement layer data. For example, the resolution of the base layer data may be 4K, and the resolution of the enhancement layer data may be 2K or 1080P. For another example, in other embodiments, the resolution of the base layer data may be less than the resolution of the enhancement layer data. For example, the resolution of the base layer data may be 1080P or 2K, and the resolution of the enhancement layer data may be 4K.

[0281] Scenario 3: Fig. 13C The third image processing flow diagram provided in this application is as follows: Fig. 13CAs shown, the decoding module uses the decoder to obtain the base layer data, the enhancement layer data and the metadata from the bit stream, and the processing module uses the metadata to perform processing 1 on the base layer data to obtain the processed base layer data. And the processing module uses the metadata to perform processing 2 on the enhancement layer data to obtain the processed enhancement layer data. The processing module synthesizes the processed base layer data and the processed enhancement layer data to obtain an HDR image (i.e., the target image). The processing module transmits the HDR image to the display module for display.

[0282] The above text, combined with the accompanying drawings, describes the process of performing image processing by the electronic device provided by the present application. In some possible situations, modules deployed on different electronic devices can also be used to implement the above image processing process. Three possible examples are given below.

[0283] Example 1, Fig.13D A schematic diagram of the fourth image processing process provided in this application is as follows: Fig.13D As shown, the decoding module and the processing module cooperate to implement the image processing process to obtain the target image, and the display module displays the target image. The decoding module decodes the received code stream to obtain the basic image, the gain map and the metadata, the processing module processes the basic image using the metadata to obtain the processed basic image, and the processing module obtains the target image according to the processed basic image and the gain map.

[0284] Example 2, Fig.13E A schematic diagram of the fifth image processing process provided in this application is as follows: Fig.13E As shown, the decoding module decodes the received code stream to obtain the base image, the enhancement layer and the metadata, and synthesizes the enhancement layer and the base layer to obtain the HDR image synthesis. The image processing module processes the HDR image synthesis 2 to obtain the processed image 1, and transmits the processed image 1 to the display module. The image processing module processes the base layer 1 to obtain the processed image 2, and transmits the processed image 2 to the display module. The display module receives the processed image 1 and the processed image 2, and obtains the target image according to the processed image 1 and the processed image 2, and displays the target image.

[0285] Example 3, Fig.13F A schematic diagram of the sixth image processing process provided in this application is as follows: Fig.13F As shown, basic layer data, metadata, and enhanced layer data are obtained according to code stream decoding, or the basic layer data, metadata, and enhanced layer data are directly obtained from the code stream. Processing 1 is performed on the basic layer data to obtain processed image 1, processing 2 is performed on the enhanced layer data to obtain processed image 2, and processed image 1 and processed image 2 are synthesized, and the synthesized image is displayed.

[0286] The above describes the general process of image processing in conjunction with the accompanying drawings. The following describes a data processing method provided by the present application in conjunction with the accompanying drawings. Fig.14 A flow chart of a data processing method provided in this application is as follows: Fig.14 As shown, the method includes the following S1410 to S1460.

[0287] S1410: The encoder determines, according to the acquired first image, base layer data of the first image, enhancement layer data of the first image, and metadata of the first image.

[0288] The metadata carries the conversion information.

[0289] In a possible scenario, the conversion information includes at least one of color mode information, chroma format information, color gamut information, and color compensation information.

[0290] The color mode information includes: color mode information of the basic layer data of the first image, color mode information of the enhanced layer data of the first image, and at least one of the color mode information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0291] The chroma format information includes: chroma format information of the basic layer data of the first image, chroma format information of the enhanced layer data of the first image, and at least one of the chroma format information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0292] The color gamut information includes: color gamut information of the basic layer data of the first image, color gamut information of the enhanced layer data of the first image, and at least one of the color gamut information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0293] The color compensation information includes: color compensation information of the basic layer data of the first image, color compensation information of the enhanced layer data of the first image, and at least one of color compensation information corresponding to the basic layer data of the first image and the enhanced layer data of the first image that needs to be transformed and processed.

[0294] In a possible scenario, the conversion information further includes: conversion position information and conversion scheme information. The conversion position information is used to indicate the position where the conversion occurs. The conversion scheme information is used to indicate the conversion scheme used when converting the target data. The conversion occurs before or after the second image is obtained. The second image is used to indicate an image obtained based on the base layer data and the enhancement layer data.

[0295] The content of the conversion information varies depending on where the conversion occurs, and is described below in different situations.

[0296] In case A, the conversion occurs before obtaining the second image.

[0297] In this case, the conversion information also includes conversion object information. The conversion object information is used to indicate the conversion object for which the conversion scheme needs to be executed. The conversion object includes: at least one of basic layer data and enhanced layer data, such as the conversion object includes basic layer data, or the conversion object includes enhanced layer data, or the conversion object includes basic layer data and enhanced layer data.

[0298] The content of the conversion plan varies depending on the conversion object, and is described below in different situations.

[0299] In case Aa, the conversion object includes basic layer data.

[0300] In this case, the conversion scheme information includes: a basic layer data conversion scheme. The basic layer data conversion scheme is used to indicate that at least one of color gamut conversion, chroma format conversion, color compensation and color mode conversion is performed on the basic layer data. Color gamut conversion of the basic layer data may refer to color gamut information in the transformation and processing corresponding to the basic layer data and the enhancement layer data carried in the metadata, or preset color gamut information is used to perform color gamut conversion on the basic layer data. Color mode conversion of the basic layer data may refer to color mode information in the transformation and processing corresponding to the basic layer data and the enhancement layer data carried in the metadata, or preset color mode information is used to perform color mode conversion on the basic layer data. Color format conversion of the basic layer data may refer to color mode conversion of the basic layer data using the chroma format information in the transformation and processing corresponding to the basic layer data and the enhancement layer data carried in the metadata, or preset chroma format information. Color compensation for base layer data may refer to color compensation for base layer data using color compensation information in transformation and processing corresponding to base layer data and enhancement layer data carried in metadata, or using preset color compensation information.

[0301] The basic layer data conversion scheme is also used to indicate conversion parameters used when converting the basic layer data. The conversion parameters include: at least one of a compensation matrix, a conversion matrix, a conversion function and a compensation value.

[0302] Case Ab, the conversion object includes enhancement layer data.

[0303] In this case, the conversion scheme information includes: an enhancement layer data conversion scheme. The enhancement layer data conversion scheme is used to indicate that at least one of chroma format conversion, color mode conversion and channel conversion is performed on the enhancement layer data.

[0304] For the relevant contents of chroma format conversion and color mode conversion, please refer to the relevant description in the above scenario Aa, which will not be repeated here. Channel conversion is used to restore the channels of the enhancement layer data to the original channels.

[0305] The enhancement layer data conversion scheme is also used to indicate conversion parameters used when converting the enhancement layer data. The conversion parameters include: at least one of a conversion matrix, a conversion function and a channel conversion function.

[0306] When the conversion object is basic layer data, the electronic device may convert the basic layer data using the basic layer data conversion scheme described in scenario Aa. When the conversion object is enhanced layer data, the electronic device may convert the enhanced layer data using the enhanced layer data conversion scheme described in scenario Ab. And when the conversion object is basic layer data and enhanced layer data, the electronic device may convert the basic layer data using the basic layer data conversion scheme described in scenario Aa, and convert the enhanced layer data using the enhanced layer data conversion scheme described in scenario Ab.

[0307] In case B, the conversion occurs after the second image is obtained.

[0308] In this case, the conversion scheme information includes: at least one of color gamut conversion, chroma format conversion, color compensation and color mode conversion. For the relevant contents of color gamut conversion, chroma format conversion, color compensation and color mode conversion, please refer to the relevant description above, which will not be repeated here.

[0309] In a possible scenario, the conversion scheme information further includes: conversion parameters. The conversion parameters include: at least one of a compensation matrix, a conversion matrix, a conversion function, and a compensation value. For the description of the compensation matrix, the conversion matrix, the conversion function, and the compensation value, please refer to the general technology, which will not be repeated here.

[0310] In one possible scenario, when the target data includes basic layer data and enhanced layer data, the conversion information may further include channel merging information. The channel merging information is used to indicate a scheme for combining channels of the basic layer data and channels of the enhanced layer data. The scheme for combining the basic layer data and the enhanced layer data varies depending on the color mode of the basic layer data and the color mode of the enhanced layer data. In this case, the channel merging information may be used to indicate a scheme for combining channels corresponding to the color mode of the basic layer data and channels corresponding to the color mode of the enhanced layer data. Depending on the color mode of the basic layer data and the color mode of the enhanced layer data, the schemes for combining the basic layer data and the enhanced layer data are different, which are described below.

[0311] When the color modes of the basic layer data and the enhanced layer data are both RGB modes, the channel merging information is used to indicate: the R channel of the basic layer data and the R channel of the enhanced layer data are merged, the G channel of the basic layer data and the G channel of the enhanced layer data are merged, and the B channel of the basic layer data and the B channel of the enhanced layer data are merged.

[0312] When the color mode of the basic layer data is the RGB mode and the color mode of the enhanced layer data is the grayscale mode, the channel merging information is used to indicate: the R channel of the basic layer data is merged with the grayscale value of the enhanced layer data, the G channel of the basic layer data is merged with the grayscale value of the enhanced layer data, and the B channel of the basic layer data is merged with the grayscale value of the enhanced layer data.

[0313] When the color mode of the basic layer data is the grayscale mode and the color mode of the enhanced layer data is the RGB mode, the channel merging information is used to indicate: the R channel of the enhanced layer data is merged with the grayscale value of the basic layer data, the G channel of the enhanced layer data is merged with the grayscale value of the basic layer data, and the B channel of the enhanced layer data is merged with the grayscale value of the basic layer data.

[0314] In the case that the color modes of the basic layer data and the enhancement layer data are both grayscale modes, the channel merging information is used to indicate that the grayscale value of the basic layer data and the grayscale value of the enhancement layer data are merged.

[0315] When the color mode of the basic layer data and the color mode of the enhanced layer data are any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0316] When the color modes of the basic layer data and the enhanced layer data are any one of the following: YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, the channel merging information is used to indicate that the brightness channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0317] When the color mode of the basic layer data is grayscale mode and the color mode of the enhanced layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, the channel merging information is used to indicate: the channel of the basic layer data and the brightness channel of the enhanced layer data are merged.

[0318] When the color mode of the basic layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, and the color mode of the enhanced layer data is the grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the channel of the enhanced layer data are merged.

[0319] There are multiple ways to merge channels, which may be preset or carried by metadata. Several possible ways to merge channels are given below.

[0320] Method 1: The channels of the base layer data and the channels of the enhancement layer data are directly multiplied.

[0321] In this case, the decoder can use formula (6) to merge the channels of the base layer data and the channels of the enhancement layer data.

[0322] newPictue[i]=Base[i]*Enhance[i] Formula (6)

[0323] Where newPictue[i] refers to the merged image. Base[i] refers to the base layer data. Enhance[i] refers to the enhanced layer data.

[0324] Method 2: The channels of the basic layer data and the channels of the enhanced layer data are directly added.

[0325] In this case, the decoder can use formula (7) to merge the channels of the base layer data and the channels of the enhancement layer data.

[0326] newPictue[i]=Base[i]+Enhance[i] Formula (7)

[0327] Where newPictue[i] refers to the merged image. Base[i] refers to the base layer data. Enhance[i] refers to the enhanced layer data.

[0328] Method 3: The channels of the basic layer data and the channels of the processed enhancement layer data are directly multiplied.

[0329] In this case, the decoder can use formula (8) to merge the channels of the base layer data and the channels of the enhancement layer data.

[0330] newPictue[i]=Base[i]*f(Enhance[i]) Formula (8)

[0331] Where newPictue[i] refers to the merged image. Base[i] refers to the base layer data. Enhance[i] refers to the enhancement layer data. f() is the value domain conversion function.

[0332] Method 4: The channels of the basic layer data and the channels of the processed enhancement layer data are directly added.

[0333] In this case, the decoder can use formula (9) to merge the channels of the base layer data and the channels of the enhancement layer data.

[0334] newPictue[i]=Base[i]+f(Enhance[i]) Formula (9)

[0335] Where newPictue[i] refers to the merged image. Base[i] refers to the base layer data. Enhance[i] refers to the enhancement layer data. f() is the value domain conversion function.

[0336] Mode 5: directly multiply the channels of the base layer data and the channels of the enhancement layer data after the first processing, and perform the second processing on the obtained results.

[0337] In this case, the decoder can use formula (10) to merge the channels of the base layer data and the channels of the enhancement layer data.

[0338] newPictue[i]=g(Base[i]*f(Enhance[i])) Formula (10)

[0339] Where newPictue[i] refers to the merged image. Base[i] refers to the base layer data. Enhance[i] refers to the enhancement layer data. f() is the value domain conversion function, and g() is the mapping function.

[0340] Method 6: directly add the channels of the basic layer data and the channels of the enhanced layer data after the first processing, and perform the second processing on the obtained results.

[0341] In this case, the decoder can use formula (11) to merge the channels of the base layer data and the channels of the enhancement layer data.

[0342] newPictue[i]=g(Base[i]+f(Enhance[i])) Formula (11)

[0343] Where newPictue[i] refers to the merged image. Base[i] refers to the base layer data. Enhance[i] refers to the enhancement layer data. f() is the value domain conversion function, and g() is the mapping function.

[0344] According to the needs of practical applications, the value domain conversion function f() and the mapping function g() in Methods 1 to 6 can be any functions, and this application does not limit them, such as f(x)=x, g(x)=x.

[0345] The following takes the channel merging method as the multiplication described in method 1 and the addition described in method 2 as an example to specifically describe the scheme for merging the channel of the basic layer data and the channel of the enhancement layer data. The channel merging method may also be other merging methods, such as the methods described in methods 3 to 6 above, etc. According to the needs of actual applications, the channel merging method may also be other methods other than methods 1 to 6, and this application is not limited to this.

[0346] According to the difference between the number of channels of the basic layer data and the number of channels of the enhanced layer data, the channel merging methods are different, which are described below respectively.

[0347] Case 1: the base layer data is a single channel, and the enhancement layer data is a single channel.

[0348] In this case, the channel merging information is used to indicate that the single channel of the base layer data and the single channel of the enhancement layer data are merged in the channel merging manner described above. Exemplarily, the channel of the base layer data and the channel of the enhancement layer data are directly merged. Taking the merge processing as the manner given in Example 1 and Example 2 as an example, the content indicated by the channel merging information may be the content in Table 1.

[0349] Table 1

[0350] Channels of Image 1 Channels of Image 2 Channel merging information Channel C1 of image 2 Channel C2 of image 2 (C1+C2) / C1*C2

[0351] When image 1 is basic layer data, image 2 may be enhanced layer data, or when image 1 is enhanced layer data, image 2 may be basic layer data.

[0352] Case 2: the base layer data is multi-channel and the enhancement layer data is single-channel.

[0353] In this case, the channel combination scheme indicated by the channel merging information is different according to the different color modes of the basic layer data.

[0354] (1) A color mode in which the brightness and chrominance of the base layer data are not separated.

[0355] In this case, the channel merging information indicates: merging the channels of the enhancement layer data with the channels of the base layer data. Taking the base layer data in RGB mode as an example, the content indicated by the channel merging information may be the content in Table 2.

[0356] Table 2

[0357] Channels of Image 1 Channels of Image 1 Channel merging information Channels of Image 1 (R, G, B) Channel C of Image 2 (R+C, G+C, B+C) / (R*C, G*C, B*C)

[0358] When image 1 is basic layer data, image 2 may be enhanced layer data.

[0359] (2) The basic layer data is a color model in which brightness and chrominance are separated.

[0360] In this case, the channel combining scheme indicated by the channel merging information is different depending on whether the channel of the enhancement layer data is used to enhance the channels of the basic layer data other than the brightness channel (such as the chroma channel, the hue channel, etc.).

[0361] ① Use the channels of the enhanced layer data to process the channels of the basic layer data except the brightness channel.

[0362] In this case, the channel merging information indicates that the channel of the enhancement layer data is merged into the brightness channel of the base layer data and at least one other channel except the brightness channel.

[0363] For example, the channel merging information indicates: merge the channel of the enhancement layer data into each channel of the basic layer data. Taking the basic layer data as YUV mode and the channel of the enhancement layer data as C as an example, the channel merging information indicates that the channel C of the enhancement layer data is merged into the Y channel, U channel and V channel of the basic layer data, which can be represented by the content corresponding to the row number 1 in Table 3.

[0364] For another example, the channel merging information indicates: merging the channel of the enhancement layer data into the luminance channel and chrominance channel of the base layer data. Taking the base layer data in YUV mode and the channel of the enhancement layer data as C as an example, the channel merging information indicates merging the channel C of the enhancement layer data into the Y channel and U channel of the base layer data, which can be represented by the content corresponding to the row number 2 in Table 3.

[0365] ② Use the channel of the enhancement layer data to process the brightness channel of the basic layer data.

[0366] The channel merging information indicates: merging the channel of the enhancement layer data into the brightness channel of the base layer data. For example, the base layer data is in YUV mode and the channel of the enhancement layer data is C. In this case, the content indicated by the channel merging information may be the content corresponding to the row number 3 in Table 3.

[0367] Table 3

[0368]

[0369] When image 1 is basic layer data, image 2 may be enhanced layer data.

[0370] Case 3: the base layer data is multi-channel and the enhancement layer data is multi-channel.

[0371] In this case, the base layer data and the enhancement layer data may both be color modes in which brightness and chroma are separated, or both are color modes in which brightness and chroma are not separated, or one is a color mode in which brightness and chroma are separated and the other is not a color mode in which brightness and chroma are not separated. In different cases, the channel merging information is different, which is explained below respectively.

[0372] (1) Both the base layer data and the enhancement layer data are not in a mode in which luminance and chrominance are not separated.

[0373] The basic layer data and the enhanced layer data can be in the same color mode or in two different color modes, which are described below with examples.

[0374] ① The basic layer data and the enhanced layer data use the same color mode.

[0375] In this case, the channel merging information indicates that at least one channel of the enhancement layer data is merged with the corresponding channel of the base layer data. For example, if the channel merging information indicates that each channel of the enhancement layer data is merged with the corresponding channel of the base layer data, and both the base layer data and the enhancement layer data are in RGB mode, RGBA mode, or CMYK mode, the content indicated by the channel merging information may be the content in Table 4.

[0376] Table 4

[0377]

[0378] When image 1 is basic layer data, image 2 may be enhanced layer data.

[0379] In the above, the example of merging each channel of the basic layer data and the corresponding channel of the enhancement layer data is taken. In some possible situations, the channel merging information may also indicate that some channels of the enhancement layer data and the corresponding channels of the basic layer data are to be merged.

[0380] For example, the basic layer data and the enhanced layer data are in RGBA mode, and the channel merging information indicates: the R channel of the basic layer data and the R channel of the enhanced layer data are merged, the G channel of the basic layer data and the G channel of the enhanced layer data are merged, and the B channel of the basic layer data and the R channel of the enhanced layer data are merged.

[0381] ② There are two different color modes for basic layer data and enhanced layer data.

[0382] In this case, the channel merging information indicates that each channel of the enhanced layer data is merged with the corresponding channel of the base layer data. For example, the base layer data is in RGB mode and the enhanced layer data is in RGBA mode. In this case, the content indicated by the channel merging information may be the content in Table 5.

[0383] Table 5

[0384]

[0385] When image 1 is basic layer data, image 2 may be enhanced layer data, or when image 1 is enhanced layer data, image 2 may be basic layer data.

[0386] (2) The base layer data and the enhancement layer data are both in color modes with brightness and chrominance separated.

[0387] The channel merging information indicates at least one of merging the luminance channel of the enhancement layer data and the luminance channel of the base layer data, and merging the chrominance channel of the enhancement layer data and the chrominance channel of the base layer data.

[0388] ① The basic layer data and the enhanced layer data use the same color mode.

[0389] In this case, the channel merging information may indicate that the brightness channel of the enhancement layer data and the brightness channel of the base layer data are merged. For example, if the color modes of the base layer data and the enhancement layer data are both YUV mode, Lab mode, or HSV mode, and the channel merging information indicates that only the brightness channel of the enhancement layer data and the brightness channel of the base layer data are merged, in this case, the channel merging information may be represented by the content in Table 6.

[0390] Table 6

[0391]

[0392] The above takes the example of channel merging information indicating the merging of the brightness channel of the enhancement layer data and the brightness channel of the basic layer data. In some possible situations, the channel merging information can also indicate: the brightness channel of the enhancement layer data and at least one of the other channels are merged with the corresponding channel of the basic layer data.

[0393] For example, the color modes of the basic layer data and the enhanced layer data are both YUV modes, and the channel merging information can indicate that the Y channel of the enhanced layer data is merged into the Y channel of the basic layer data, the U channel of the enhanced layer data is merged into the U channel of the basic layer data, and the V channel of the enhanced layer data is merged into the V channel of the basic layer data.

[0394] For example, the color modes of the basic layer data and the enhanced layer data are both Lab modes, and the channel merging information can indicate that the L channel of the enhanced layer data is merged into the L channel of the basic layer data, the a channel of the enhanced layer data is merged into the a channel of the basic layer data, and the b channel of the enhanced layer data is merged into the b channel of the basic layer data.

[0395] For another example, the color modes of the basic layer data and the enhanced layer data are both HSV modes, and the channel merging information can indicate that the H channel of the enhanced layer data is merged into the H channel of the basic layer data, the S channel of the enhanced layer data is merged into the S channel of the basic layer data, and the V channel of the enhanced layer data is merged into the V channel of the basic layer data.

[0396] ② There are two color modes for basic layer data and enhanced layer data.

[0397] For example, the basic layer data is in YUV mode, and the enhanced layer data is in Lab mode. For another example, the basic layer data is in HSV mode, and the enhanced layer data is in Lab mode. For another example, the basic layer data is in YUV mode, and the enhanced layer data is in HSV mode. In this case, it is necessary to convert the color modes of the basic layer data and the enhanced layer data, and merge the channels of the basic layer data and the enhanced layer data after the color mode conversion. For example, the color mode of the basic layer data and the color mode of the enhanced layer data are both converted into RGB mode to obtain the converted basic layer data in RGB mode and the enhanced layer data in RGB mode. In this case, the channel merging information corresponding to sequence number 1 in Table 4 above can be used to merge the converted basic layer data and the converted enhanced layer data. (3) One of the basic layer data and the enhanced layer data is in a color mode in which brightness and chrominance are separated, and the other is in a color mode in which brightness and chrominance are not separated.

[0398] In this case, the channel merging information indicates: using the conversion information to perform color mode conversion on at least one of the base layer data and the enhancement layer data, and using the methods described in (1) and (2) above to perform channel merging on the converted data. For the same contents of the channel merging information of the converted base layer data and the enhancement layer data, please refer to the relevant descriptions in (1) and (2) above, which will not be repeated here.

[0399] S1420, the encoder encodes the base layer data, the enhancement layer data and the metadata to obtain a code stream.

[0400] Metadata can be used to define the aspect ratio of an image. The resolution of an image does not necessarily represent the aspect ratio. If the distribution of pixels on the display end is not spatially uniform, that is, the horizontal and vertical distances between pixels are different, this may cause errors in the phases of different interpolation algorithms. For example, the actual distances between adjacent horizontal and vertical pixels that differ by 1 / 2 are different. Metadata is used to determine the range of actual values ​​stored in the image, or the upper and lower limits of the dynamic range. For example: when reading an HDR image during image input and output, it can be 10 bits wide, but the image belongs to the limited range. This limited range is specified in documents such as BT.2020 as Luma is 64 to 940 and Chroma is 64 to 960 (for 10 bits). In the example, the upper and lower ranges of this bit width should be indicated by metadata, and the larger bit width in the enhancement layer data or the base layer image should be taken. The corresponding is the full range, which is 0 to 1023 in 10 bits.

[0401] S1430, the encoder transmits the code stream to the decoder.

[0402] Corresponding to S1430, the decoder executes S1440. At S1440, the decoder receives a bitstream and obtains metadata from the bitstream.

[0403] S1450: The decoder determines target data according to the conversion information carried in the metadata.

[0404] The metadata is metadata of the first image, the target data includes basic layer data of the first image and / or enhanced layer data of the first image; or, the target data includes the second image.

[0405] The decoder determines the target data according to the conversion position information of the conversion information carried in the metadata.

[0406] (1) If the conversion position information indicates that the conversion occurs after the second picture is obtained, the decoder determines that the target data is the second picture.

[0407] (2) If the conversion position information indicates that the conversion occurs before the second image is obtained, the decoder determines that the target data is at least one of the base layer data and the enhancement layer data. In this case, the decoder further determines that the conversion object is the base layer data, the enhancement layer data, or the base layer data and the enhancement layer data according to the conversion object information of the conversion information carried in the metadata.

[0408] S1460: The decoder converts the target data according to the conversion information.

[0409] Depending on the target data, the process of converting the target data by the decoder is different, which is described below in different cases.

[0410] In the first possible case, the target data is base layer data.

[0411] Fig.15 A schematic diagram of the first conversion process provided for this application is shown in FIG. Fig.15 As shown, the decoder converts the base layer data according to the conversion information in the metadata to obtain the converted base layer data. The conversion may include, but is not limited to: color compensation, color gamut conversion, chroma format conversion and color mode conversion. The decoder converts the color gamut information, chroma format and color mode of the base layer data according to the color gamut information, chroma format and color mode in the process of combining the base layer data and the enhancement layer data carried in the metadata. The following describes the conversion process of the decoder on the base layer data.

[0412] (1) For color compensation.

[0413] The decoder may use a compensation matrix, a compensation value, etc. to perform color supplement on the base layer data to obtain compensated base layer data. The compensation matrix, the supplement value, etc. may be preset or carried in metadata, and this application is not limited thereto. Exemplarily, the decoder uses the compensation value carried in the metadata to perform color supplement on the base layer data to obtain color compensated base layer data.

[0414] (2) Regarding color gamut conversion.

[0415] The color gamut information of the base layer data is the first color gamut information, and the color gamut information in the process of combining the base layer data with the enhancement layer data is the second color gamut information. The color gamut information may refer to BT.2020, DCI-P3, Rec.709, NTSC, Rec.2020, etc. Depending on whether the first color gamut information and the second color gamut information are the same, the decoder processes the color gamut information of the base layer data differently, which are described below.

[0416] ① The first color gamut information is the same as the second color gamut information.

[0417] In this case, the decoder may not convert the color gamut information of the base layer data, or may perform an identity transformation on the color gamut information of the base layer data. The manner in which the decoder performs an identity transformation on the color gamut information of the base layer data may be carried by metadata or may be preset.

[0418] ② The first color gamut information is different from the second color gamut information.

[0419] In this case, the decoder converts the color gamut information of the basic layer data (i.e., the first color gamut information) into the color gamut information in the process of combining the basic layer data with the enhanced layer data (i.e., the second color gamut information) according to the color gamut information of the basic layer data (or the first color gamut information) and the color gamut information in the process of combining the basic layer data with the enhanced layer data (or the second color gamut information). The second color gamut information may be preset color gamut information or color gamut information carried in metadata. The decoder may convert the color gamut information of the basic layer data from the first color gamut information to the second color gamut information in a variety of ways, such as a conversion matrix, a conversion function, and the like. The conversion matrix or the conversion function may be a default one or may be carried in the metadata, which is not limited in this application. Exemplarily, the decoder converts the color gamut information of the basic layer data from the first color gamut information to the second color gamut information using a 3*3 conversion matrix carried in the metadata.

[0420] (3) Regarding color format conversion.

[0421] The chroma format of the base layer data is the first chroma format, and the chroma format in the process of combining the base layer data with the enhancement layer data is the second chroma format. The chroma format may refer to R:G:B, YUV4:2:0, YUV4:4:4, YUV4:2:2, etc. Depending on whether the first chroma format is the same as the second chroma format, the decoder processes the chroma format of the base layer data differently, which are described below.

[0422] ① The first chroma format is the same as the second chroma format.

[0423] In this case, the decoder may not convert the chrominance format of the base layer data, or may perform an identical transformation on the chrominance format of the base layer data. The manner in which the decoder performs an identical transformation on the chrominance format of the base layer data may be carried by metadata or may be preset.

[0424] ②The first chroma format is different from the second chroma format.

[0425] The decoder converts the chroma format of the base layer data (i.e., the first chroma format) into the chroma format in the process of combining the base layer data with the enhancement layer data (i.e., the second chroma format) according to the chroma format of the base layer data (i.e., the first chroma format) and the chroma format in the process of combining the base layer data with the enhancement layer data (i.e., the second chroma format). The second chroma format may be a preset chroma format or a chroma format carried in metadata.

[0426] The chroma format of the base layer data and the chroma format in the process of combining the base layer data with the enhancement layer data are different, and the decoder converts the chroma format in different ways, which are described below.

[0427] Mode 1: If the first chroma format is YUV4:2:0 or YUV4:2:2, and the second chroma format is R:G:B. The decoder needs to convert the chroma format of the base layer data twice, so that the chroma format of the base layer data after the two chroma conversions is the same as the chroma format of the enhancement layer data and is R:G:B. The first conversion of the chroma format of the base layer data by the decoder is: converting the chroma format of the base layer data from YUV4:2:0 or YUV4:2:2 to YUV4:4:4. The second conversion of the chroma format of the base layer data by the decoder is: converting the chroma format of the base layer data obtained by the first conversion from YUV4:4:4 to R:G:B. The decoder can use multiple methods to convert the chroma format of the base layer data from YUV4:2:0 or YUV4:2:2 to YUV4:4:4, such as the decoder uses upsampling to convert the chroma format of the base layer data from YUV4:2:0 or YUV4:2:2 to YUV4:4:4. The decoder can convert the chrominance format of the base layer data from YUV4:4:4 to R:G:B in a variety of ways, such as conversion matrix, conversion function, etc. The conversion matrix, conversion function, etc. can be preset or carried in metadata, and this application does not limit this.

[0428] Mode 2: If the first chroma format is YUV4:4:4 and the second chroma format is R:G:B, the decoder converts the chroma format of the base layer data once, so that the chroma format of the converted base layer data is the same as the chroma format of the enhancement layer data and is R:G:B. The decoder can convert the chroma format of the base layer data from YUV4:4:4 to R:G:B in a variety of ways, such as conversion matrix, conversion function, etc. The conversion process in this case is the same as the conversion process in Mode 1.

[0429] Mode 3, if the first chroma format is R:G:B, and the second chroma format is YUV4:2:0 or YUV4:2:2. The decoder needs to convert the chroma format of the base layer data twice, so that the chroma format of the base layer data after the two chroma conversions is the same as the chroma format of the enhancement layer data and is YUV4:2:0 or YUV4:2:2. The first conversion of the chroma format of the base layer data by the decoder is: converting the chroma format of the base layer data from R:G:B to YUV4:4:4. The second conversion of the chroma format of the base layer data by the decoder is: converting the chroma format of the base layer data obtained by the first conversion from YUV4:4:4 to YUV4:2:0 or YUV4:2:2. The decoder can use a variety of methods to convert the chroma format of the base layer data from R:G:B to YUV4:4:4, such as conversion matrix, conversion function, etc. The conversion process in this case is the inverse process of the conversion process in mode 1. The decoder may convert the chroma format of the base layer data from YUV4:4:4 to YUV4:2:0 or YUV4:2:2 in a variety of ways, such as converting the chroma format of the base layer data from YUV4:4:4 to YUV4:2:0 or YUV4:2:2 by downsampling.

[0430] Mode 4: If the first chroma format is R:G:B and the second chroma format is YUV4:4:4, the decoder converts the chroma format of the base layer data once, so that the chroma format of the converted base layer data is the same as the chroma format of the enhancement layer data and is YUV4:4:4. The decoder can convert the chroma format of the base layer data from R:G:B to YUV4:4:4 in a variety of ways, such as conversion matrix, conversion function, etc. The conversion process in this case is the inverse process of the conversion process in Mode 1.

[0431] (4) Regarding color mode conversion.

[0432] The color mode of the base layer data is the first color mode, and the color mode in the process of combining the base layer data with the enhancement layer data is the second color mode. The color mode may refer to RGB, YUV, Lab, HSV, RGBA, CMYK, etc. Depending on whether the first color mode and the second color mode are the same, the decoder processes the color mode of the base layer data differently, which are described below.

[0433] ①The first color mode is the same as the second color mode.

[0434] In this case, the decoder may not convert the color mode of the base layer data.

[0435] ②The first color mode is different from the second color mode.

[0436] The decoder converts the color mode of the basic layer data (i.e., the first color mode) into the color mode in the process of combining the basic layer data with the enhanced layer data (i.e., the second color mode) according to the color mode of the basic layer data (or the first color mode) and the color mode in the process of combining the basic layer data with the enhanced layer data (or the second color mode). The second color mode can be a preset color mode or a color mode carried in the metadata. The decoder can convert the color mode of the basic layer data from the first color mode to the second color mode in a variety of ways, such as a conversion matrix, a conversion function, etc., which is not limited in this application. Exemplarily, the decoder uses a 3*3 conversion matrix carried in the metadata to convert the color mode of the basic layer data from the first color mode to the second color mode.

[0437] In the second possible scenario, the target data is enhancement layer data.

[0438] Fig.16 A schematic diagram of the second conversion process provided for this application is shown in FIG. Fig.16 As shown, the decoder converts the enhancement layer data according to the conversion information in the metadata to obtain the converted enhancement layer data. The conversion may include, but is not limited to: color compensation, color gamut conversion, chroma format conversion and color mode conversion. The methods for the decoder to perform color compensation, color gamut conversion, chroma format conversion and color mode conversion on the enhancement layer data are respectively the same as the methods for color compensation, color gamut conversion, chroma format conversion and color mode conversion in the first possible scenario above. For relevant descriptions, please refer to the above and will not be repeated here.

[0439] In a third possible scenario, the target data is base layer data and enhancement layer data.

[0440] Fig.17 A schematic diagram of the third conversion process provided for this application is shown in FIG. Fig.17 As shown, the decoder converts the base layer data and the enhancement layer data at the same time according to the conversion information in the metadata to obtain the converted base layer data and the converted enhancement layer data. For the method of the decoder converting the base layer data, please refer to the relevant description in the first possible scenario above, and for the method of the decoder converting the enhancement layer data, please refer to the relevant description in the second possible scenario above, which will not be repeated here.

[0441] In a fourth possible situation, the target data is the second image.

[0442] Fig.18 A schematic diagram of the fourth conversion process provided by this application, such as Fig.18As shown, the decoder converts the second image according to the conversion information carried in the metadata to obtain the converted second image. The conversion may include, but is not limited to: color compensation, color gamut conversion, chroma format conversion and color mode conversion. Different from the above: the decoder converts the color information, color gamut conversion, chroma format conversion and color mode conversion of the second image according to the color information, color gamut information, chroma format and color mode used in the display process carried in the metadata. The method by which the decoder converts the color information, color gamut information, chroma format and color mode of the second image is the same as the method of color compensation, color gamut conversion, chroma format conversion and color mode conversion in the first possible scenario above. Please refer to the above for the relevant description, which will not be repeated here.

[0443] In some possible situations, the decoder may also map at least one of the base layer data, the converted base layer data, the enhanced layer data, the converted enhanced layer data, the second image, and the converted second image. The decoder may process at least one of the base layer data, the converted base layer data, the enhanced layer data, the converted enhanced layer data, the second image, and the converted second image in a variety of ways. The processing method may be preset or carried in metadata, and this application is not limited to this. The processing method may include, but is not limited to: normalization, inverse normalization, increasing offset, decreasing offset, multiplying by coefficients, multiplying by coefficients, and the like. Taking the decoder processing data and obtaining the processed dataAfter as an example, the following processing methods are explained. The data is used to indicate at least one of the base layer data, the converted base layer data, the enhanced layer data, the converted enhanced layer data, the second image, and the converted second image.

[0444] Method ①: identity transformation. The decoder can use formula (12) to perform identity transformation on data.

[0445] dataAfter[i]=data[i] Formula (12)

[0446] Where dataAfter[i] is the image after identity transformation and data[i] is the image before transformation.

[0447] Method ②: If the metadata carries the upper limit THH and lower limit THL of the data, the decoder can use formula (13) to process the data.

[0448] dataAfter[i] =data[i]* THH+(A-data[i])*THL formula (13)

[0449] Where dataAfter[i] is the result of processing. data[i] is the image before processing. A is the maximum value of the stored data. When data is normalized to between 0 and 1.0, A is 1.0.

[0450] Method ③: If the metadata carries the upper limit THH and lower limit THL of data, data can be non-linear coded data such as PQ / HLG / LOG / Gamma. First, data[i] is converted to the linear domain, and then formula (13) is used to process data[i] converted to the linear domain.

[0451] Method ④: If the metadata carries the lower limit THL of data, the decoder can use formula (14) to process data.

[0452] dataAfter[i]=data[i]+THL formula (14)

[0453] Among them, dataAfter[i] is the result obtained by processing, and data[i] is the image before processing.

[0454] Method ⑤: If the metadata carries the upper limit THH of data, the decoder can use formula (15) to process data.

[0455] dataAfter[i]=THH+A-data[i] formula (15)

[0456] Where dataAfter[i] is the result of processing. data[i] is the image before processing. A is the maximum value of the stored data. When data is normalized to between 0 and 1.0, A is 1.0.

[0457] Method ⑥: If the metadata contains processing parameters, the decoder can obtain the processing method TME() according to the parameters and use formula (16) to process the data.

[0458] dataAfter[i]=TME(data[i]) Formula (16)

[0459] Wherein, dataAfter[i] is the result obtained by processing. data[i] is the image before processing. The processing method TME() can have various forms, such as Sigmoid, cubic spline, gamma, straight line or segmented curve or its inverse function, etc., which is not limited in this application. For example, the processing method TME() is the curve represented by formula (17) or its inverse curve.

[0460]

[0461] Wherein, L and L' can be normalized optical signals or electrical signals, a, b, p, and m are all photoelectric transfer coefficients, and a is equal to 1.12762, b is equal to -0.12762, p is equal to 2.3, m is equal to 0.14, and n is a set parameter. The processing method can also be a transformation in the form of a neural network. In this case, the decoder can construct a neural network based on historical data (such as historical enhancement layer data, historical converted enhancement layer data, historical basic layer data, historical converted basic layer data, etc.), and use the neural network to obtain rules for processing enhancement layer data, converted enhancement layer data, basic layer data, converted basic layer data, etc.

[0462] Method ⑦: If the metadata contains processing parameters, the decoder can obtain the processing method TME() according to the parameters and use formula (18) to process the data.

[0463] dataAfter[i]=TME()*data[i] Formula (18)

[0464] Among them, dataAfter[i] is the result obtained by processing, and data[i] is the image before processing.

[0465] In the above methods ① to ⑦, i can be a coordinate point. For example, for any coordinate point A (x, y) in a two-dimensional image, i is the coordinate point of the coordinate point A in the one-dimensional image after the two-dimensional image is transformed into a one-dimensional image.

[0466] In one possible scenario, if the target data includes base layer data and enhancement layer data, the decoder may also merge the channels of the base layer data and the enhancement layer data according to the channel merging information. The channel merging information is used to indicate how the channels of the base layer data are combined with the channels of the enhancement layer data. The base layer data may have multiple channels. The enhancement layer data may have a single channel or multiple channels. The channel merging information may be preset or carried in metadata. The base layer data and the enhancement layer data may refer to images directly obtained by the decoder parsing the code stream, or images obtained by the decoder after conversion using conversion information, or images obtained by the decoder processing images obtained by parsing the code stream, and this application does not limit this.

[0467] Different color modes may contain different numbers of channels. For example, the YUV color mode contains three channels, Y, U, and V, the grayscale color mode contains only one channel, and the CMYK color mode contains four channels. The number of channels contained in the base layer data and the enhancement layer data is different, and the decoder combines the channels of the base layer data and the enhancement layer data in different ways, which are described below.

[0468] Case ①: both the base layer data and the enhancement layer data are single-channel.

[0469] In this case, the decoder can merge the channels of the base layer data and the channels of the enhancement layer data according to the channel merging information described in the above case 1, such as directly adding or multiplying the channels of the base layer data and the channels of the enhancement layer data.

[0470] Case ②, the base layer data includes multiple channels, and the enhancement layer data includes a single channel.

[0471] In this case, the decoder can merge the channels of the base layer data and the channels of the enhancement layer data according to the channel merging information described in the above case 2. Depending on whether the color mode of the base layer data is a luminance and chrominance separation mode, the decoder merges the channels of the base layer data and the channels of the enhancement layer data in different ways, which are described below.

[0472] 1 The color mode for the base layer data is a color mode in which brightness and chrominance are not separated.

[0473] In this case, the decoder merges the single channel of the enhancement layer data into each channel of the base layer data according to the channel merging information described in (1) of the above case 2 to obtain an image after merging the channels. Exemplarily, the color mode of the base layer data is RGB, and the enhancement layer data is a single channel image C. In this case, the decoder performs R+C, G+C, B+C or R*C, G*C, B*C to merge the channels of the base layer data and the channels of the enhancement layer data.

[0474] 2 The color mode for the base layer data is the case where the luminance and chrominance are separated.

[0475] In this case, the decoder can merge the single channel of the enhancement layer data with the brightness channel of the base layer data according to the channel merging information described in (2) of the above case 2 to obtain an image after merging the channels. For example, the base layer data is a three-channel image with brightness and chrominance separated (such as YUV, Lab, HSV, etc.), and the enhancement layer data is a single-channel image C. The decoder merges the base layer data and the enhancement layer data according to the channel merging information described in ① in (2). In this case, the decoder executes Y*C, U*C, V*C or Y+C, U+C, V+C to achieve the merging of the channels of the base layer data and the channels of the enhancement layer data. The decoder merges the base layer data and the enhancement layer data according to the channel merging information described in ② in (2). In this case, the decoder executes Y*C, U*1, V*1 or Y+C, U+0, V+0 to achieve the merging of the channels of the base layer data and the channels of the enhancement layer data.

[0476] Case ③: both the basic layer data and the enhanced layer data are multi-channel.

[0477] In this case, depending on whether the color modes of the base layer data and the enhancement layer data are the same, the decoder combines the channels of the base layer data and the channels of the enhancement layer data in different ways, which are described below.

[0478] 1 For the case where the color modes of the base layer data and the enhancement layer data are the same.

[0479] In this case, the decoder can merge the corresponding channels of the base layer data and the enhancement layer data according to the channel merging information described in (1) of the above case 3. If the base layer data and the enhancement layer data are both in RGB mode, the decoder performs R*R, G*G, B*B or R+R, G+G, B+B on the base layer data and the enhancement layer data to merge the channels of the base layer data and the channels of the enhancement layer data. If the base layer data and the enhancement layer data are both YUV, the decoder performs Y*Y, U*U, V*V or Y+Y, U+U, V+V on the base layer data and the enhancement layer data to merge the channels of the base layer data and the channels of the enhancement layer data.

[0480] 2 For the case where one of the basic layer data and the enhanced layer data is in a color mode in which brightness and chrominance are separated, and the other is not in a color mode in which brightness and chrominance are separated, and the number of channels is the same.

[0481] In this case, the decoder needs to convert the color mode of at least one of the basic layer data and the enhanced layer data, and perform channel merging on the converted data using the merging method described above. The decoder can determine to convert the color mode of the basic layer data or the color mode of the enhanced layer data or simultaneously convert the color mode of the basic layer data and the color mode of the enhanced layer data based on the color mode in the combined processing of the basic layer data and the enhanced layer data carried in the metadata. For example, the basic layer data is in RGB mode and the enhanced layer data is in YUV mode. The color mode in the combined processing of the basic layer data and the enhanced layer data carried in the metadata is RGB. In this case, the decoder can convert the color mode of the enhanced layer data from YUV to RGB, and the decoder merges the channels of the two converted data using the method described in 1 above.

[0482] 3. For the case where one of the basic layer data and the enhanced layer data is in a color mode in which brightness and chrominance are separated, and the other is not in a color mode in which brightness and chrominance are separated, and the number of channels of the basic layer data and the enhanced layer data are different.

[0483] In this case, the decoder needs to convert the color mode of at least one of the basic layer data and the enhanced layer data, and merge the corresponding channels of the converted data. The decoder can determine to convert the color mode of the basic layer data or the color mode of the enhanced layer data or simultaneously convert the color mode of the basic layer data and the color mode of the enhanced layer data according to the color mode in the combined processing of the basic layer data and the enhanced layer data carried in the metadata. For example, the basic layer data is in RGBA mode and the enhanced layer data is in YUV mode. The color mode in the combined processing of the basic layer data and the enhanced layer data carried in the metadata is RGB. In this case, the decoder converts the enhanced layer data from YUV to RGB, and performs R*R, G*G, B*B or R+R, G+G, B+B on the basic layer data and the converted enhanced layer data, so as to realize the merging of the channels of the basic layer data and the channels of the enhanced layer data.

[0484] 4 For the case where neither the base layer data nor the enhancement layer data is in a color mode with separated luminance and chrominance and the number of channels is different.

[0485] In this case, the decoder directly merges the corresponding channels of the base layer data and the enhancement layer data to merge the channels of the base layer data and the enhancement layer data. For example, if the base layer data is in RGBA mode and the enhancement layer data is in RGB mode, the decoder performs the following operations on the base layer data and the enhancement layer data: R*R, G*G, B*B or R+R, G+G, B+B to merge the channels of the base layer data and the enhancement layer data.

[0486] In the present application, the encoder obtains the basic layer data, enhancement layer data and metadata of the first image, and encodes the basic layer data, enhancement layer data and metadata to obtain a code stream. The decoder receives and parses the code stream to obtain the basic layer data, enhancement layer data and metadata of the first image. The decoder determines and converts the target data (i.e., the basic layer data, enhancement layer data or the image obtained by merging the basic layer data and the enhancement layer data) according to the conversion information carried by the metadata. In this way, the decoder determines and converts the target data according to the conversion information carried in the metadata, so that the conversion of the target data matches the target data, and the decoder converts the target data using the conversion information matching the target data, so that the dynamic range of the converted image can not only be more highly adapted to the dynamic range supported by the display device, but also improve the display effect.

[0487] In the present application, the first basic layer data may also be referred to as the basic layer data of the first image. The first enhanced layer data may also be referred to as the enhanced layer data of the first image. The first metadata may also be referred to as the metadata or metadata of the first image. The second image may also be referred to as the new HDR image.

[0488] The embodiment of the present application also provides a method for processing a signal, comprising: obtaining first basic layer data, first enhanced layer data and first metadata. The first basic layer data is the basic layer data of a first image, the first metadata is the metadata of the first image, and the first enhanced layer data is the metadata of the first image. According to the conversion information carried in the first metadata, the target data is determined. The target data is the first basic layer data and / or the first enhanced layer data, or the target data is a second image. The second image is an image determined based on the first basic layer data and the first enhanced layer data. According to the conversion information, the target data is converted.

[0489] In a possible scenario, the first image and the second image are HDR or SDR images, wherein the first metadata may be preset or carried in a bitstream.

[0490] In a possible scenario, the conversion information is conversion indication information, and the conversion indication information includes conversion position information. When the conversion position is before the second image is determined, the conversion indication information also includes conversion object information, and the conversion object information is used to indicate that the target data includes:

[0491] the first base layer data; or

[0492] the first enhancement layer data; or

[0493] The first base layer data and the first enhancement layer data.

[0494] In a possible scenario, when the target data includes the first basic layer data, the conversion scheme information includes: a basic layer data conversion scheme, where the basic layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the first basic layer data:

[0495] Color gamut conversion;

[0496] Color compensation; or

[0497] Color space conversion.

[0498] In a possible scenario, the base layer data conversion scheme is further used to indicate conversion parameters used when converting the first base layer data.

[0499] In a possible scenario, when the base layer data conversion scheme indicates to perform the color compensation on the first base layer data, the conversion parameter includes any one of the following parameters:

[0500] Color compensation matrix;

[0501] a conversion function; or

[0502] Compensation value and compensation method.

[0503] In a possible scenario, the conversion information is conversion indication information, and the conversion indication information includes conversion scheme information. When the target data includes the first enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, and the enhancement layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the first enhancement layer data:

[0504] Color space conversion;

[0505] Channel conversion, where the channel conversion is used to restore the channels of the first enhancement layer data to the original channels.

[0506] In a possible scenario, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the first enhancement layer data.

[0507] In a possible scenario, when the target data is the second image, the conversion scheme information includes: an image conversion scheme, where the image conversion scheme is used to indicate that at least one of the following conversions is performed on the second image:

[0508] Color gamut conversion; or

[0509] Color compensation.

[0510] The embodiment of the present application also provides a coding method, the method comprising: determining first base layer data, first enhancement layer data and first metadata according to the acquired first image, wherein the conversion information is carried in the first metadata, and encoding the first base layer data, the first enhancement layer data and the first metadata to obtain a code stream.

[0511] In a possible scenario, the conversion information is conversion indication information, and the conversion indication information includes conversion position information and conversion scheme information, wherein the conversion position information is used to indicate the location where the conversion scheme occurs, and the conversion scheme information is used to indicate the conversion scheme. The first metadata may be preset or carried in the bitstream.

[0512] In a possible scenario, the conversion scheme occurs before or after a new HDR image is synthesized based on the received code stream.

[0513] In a possible scenario, when the location where the conversion occurs is before the synthesis of the new HDR image, the conversion indication information further includes conversion object information, where the conversion object information is used to indicate target data for which the conversion scheme needs to be executed, and the target data includes:

[0514] Base layer data; or

[0515] enhancement layer data; or

[0516] The base layer data and the enhancement layer data.

[0517] In a possible scenario, when the target data includes the base layer data, the conversion scheme information includes: a base layer data conversion scheme, where the base layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the base layer data:

[0518] Color gamut conversion; or

[0519] Color compensation; or

[0520] Color space conversion.

[0521] In a possible scenario, the base layer data conversion scheme is also used to indicate conversion parameters used when performing conversion.

[0522] In a possible scenario, when the base layer data conversion scheme indicates to perform the color compensation, the conversion parameter includes any one of the following parameters:

[0523] A color compensation matrix; or

[0524] a conversion function; or

[0525] Compensation value and compensation method.

[0526] In a possible scenario, when the target data includes the enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, where the enhancement layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the enhancement layer data:

[0527] Color space conversion;

[0528] Channel conversion, where the channel conversion is used to restore the channels of the enhanced layer data to the original channels.

[0529] In a possible scenario, the enhancement layer data conversion scheme is further used to indicate conversion parameters used when converting the first enhancement layer data.

[0530] In a possible scenario, when the conversion scheme occurs after the new HDR image is synthesized, the conversion scheme information includes: an image conversion scheme, where the image conversion scheme is used to indicate that at least one of the following conversions is performed on the new HDR image:

[0531] Color gamut conversion; or

[0532] Color compensation.

[0533] In one possible scenario, the conversion information includes at least one or more of the following:

[0534] Color mode information of the base layer of the first image; and / or

[0535] Color mode information of the enhancement data of the first image; and / or

[0536] The color mode information of the transformation and processing of the base layer and the enhanced data of the first image. When the image color space model of the base layer is the first color mode / space, and the color space model of the process of combining with the enhanced image is the second color mode / space, the color mode information of the transformation and processing of the base layer and the enhanced data of the first image includes a color mode / space model conversion method, and the first color mode / space is different from the second color mode / space.

[0537] The conversion information includes at least one or more of the following:

[0538] Color mode information of the base layer of the first image; and / or

[0539] Color mode information of the enhancement layer image of the first image; and / or

[0540] Color mode information of the transformation and processing of the base layer and the enhanced layer of the first image. When the color space model of the image of the base layer is the first color mode / space, and the color space model of the process of combining with the enhanced image is the second color mode / space, the color mode information of the transformation and processing of the base layer and the enhanced data of the first image includes a color mode / space model conversion method, and the first color mode / space is different from the second color mode / space.

[0541] The first color mode / space and the second color mode / space are one of the following:

[0542] RGB, YUV, YCbCr, Lab, HSV, or ICtCp.

[0543] This makes the conversion more flexible, and metadata can be adapted to multiple formats, which can facilitate more efficient and reasonable encoding.

[0544] In one possible scenario, the conversion information includes at least one or more of the following:

[0545] Chroma format information of the base layer of the first image; and / or

[0546] Chroma format information of the enhancement data of the first image; and / or

[0547] The chroma format information of the base layer of the first image and the enhanced data transformation and processing. When the chroma format information of the base layer is the first chroma format information, the chroma format information of the process of combining with the enhanced image is the second chroma format information, the conversion information includes the chroma format information of the base layer of the first image and the enhanced data transformation and processing, and the first chroma format information is different from the second chroma format information.

[0548] The conversion information includes at least one or more of the following:

[0549] Chroma format information of the base layer of the first image; and / or

[0550] Chroma format information of the enhancement layer image of the first image; and / or

[0551] The chroma format information of the base layer and the enhanced layer image of the first image is transformed and processed. When the chroma format information of the base layer is the first chroma format information, the chroma format information of the process of combining with the enhanced image is the second chroma format information, the conversion information includes the chroma format information of the base layer and the enhanced data of the first image, and the first chroma format information is different from the second chroma format information.

[0552] The first chroma format information and the second chroma format information are one of the following:

[0553] RGB4:4:4, grayscale, YUV4:4:4, YUV4:2:2, YUV4:2:0, two-channel mode.

[0554] This allows for a higher degree of matching of the color gamuts of different display devices, supports color space conversion, and facilitates more efficient and reasonable encoding.

[0555] In one possible scenario, the conversion information includes at least one or more of the following:

[0556] Information on how to combine the color mode multi-channel of the base layer of the first image with the color mode multi-channel or single channel of the enhancement data; or

[0557] The information on how to combine the color mode multi-channel of the base layer of the first image with the color mode multi-channel or single channel of the enhancement layer image. Thus, the conversion is more flexible and the metadata can be flexibly adapted in various ways.

[0558] The present application also provides another embodiment, which acts on the receiving and displaying side to decode and display all pixels in an image or video in SDR or HDR format. It mainly includes the following processes:

[0559] 1. Obtain the base layer image, enhancement layer image / enhancement data, and metadata from the bitstream.

[0560] 1. Acquisition of metadata.

[0561] The information can be obtained from metadata information encapsulated in the bitstream, from SEI of HEVC or VVC, user-defined NAL units or reserved package units, APP extension information encapsulated in JFIF, or data segments encapsulated in MP4, etc. Metadata information can also be obtained from other preset channels.

[0562] a) Metadata can contain color mode information.

[0563] The metadata may include the color mode information of the base layer, and / or the color mode information of the enhanced data, and / or the color mode information of the transformation and processing of the base layer and the enhanced data. The color space model or color mode information of the base layer / enhancement layer refers to whether the image of the base layer or enhancement layer is one of RGB, YUV, YCbCr, Lab, HSV and ICtCp.

[0564] The metadata may also include color mode conversion information. Assuming that the color space model of the base layer image is the first color mode / space (such as RGB), but the color space model of the process combined with the enhanced image / enhanced data is the second color mode / space (such as YUV), a color mode / space model conversion process is required. The process may be completed according to a preset conversion method or according to a color mode conversion method transmitted through metadata.

[0565] b) Metadata can contain luminance component and chrominance component format information.

[0566] The metadata may include information on the format (chroma subsampling) of the brightness component and chroma component of the base layer image and the enhancement layer image (such as RGB4:4:4, grayscale, YUV4:4:4, YUV4:2:2, YUV4:2:0, two-channel mode), which is used for transformation and processing of the base layer image and / or the enhancement layer image.

[0567] c) The metadata may include information on how to combine the color mode multi-channel of the base layer with the color mode multi-channel or single channel of the enhancement data.

[0568] The metadata includes channel information of the base layer and the enhancement layer, which is used to indicate how to merge the dual-layer multi-channel images.

[0569] 2. Acquisition of basic layer images.

[0570] The base layer image can be used as a source in any format. The corresponding transfer function can be used during encoding and transmission. The decoding end decodes according to the preset information or the relevant information in the metadata to obtain the base layer image. This process includes decoding the received code stream through any decoder (HEVC, JPEG, ProRes, HEIF, AV1F, etc.) to obtain the image data. The image data obtained in this process includes image data of any color space / mode (RGB, YUV, YCbCr, CIELAB, HSV, HSL, ICtCp, etc.). It should be noted that the present invention does not limit the code stream format. In terms of color space, it can be YUV or RGB. In terms of the bit width of the data, it can be 8 bits, 10 bits, or 12 bits, etc. In terms of the numerical domain, it can be a linear domain or a non-linear domain such as PQ / HLG / Gamma / log.

[0571] a) Color mode / space conversion of base layer image.

[0572] If the color space model of the base layer image is a first color mode / space (such as YUV), but the color space model of the process of combining the base layer and the enhanced layer image / enhanced data is a second color mode / space (such as RGB), a color mode / space model conversion process is required. According to the color space model information of the base layer image (preset or transmitted in metadata) and the color space model information of the process of combining the base layer and the enhanced layer image / enhanced data (preset or transmitted in metadata), the base layer image is transformed / converted to obtain a base layer image in a new format. The transformation / conversion method includes a default method or a description of the conversion process in metadata, and the 3x3 conversion matrix specified in the ITU standard or other methods can be used.

[0573] b) Chroma format conversion of base layer image.

[0574] If the luminance component and chrominance component formats of the base layer image are in the first chrominance format (such as YUV4:2:0), and the luminance component and chrominance component formats of the new base layer image to be processed and converted are in the second chrominance format (such as RGB4:4:4), the chrominance format of the base layer image needs to be converted during the conversion process. This includes the chrominance format upsampling conversion from YUV4:2:0 to YUV4:4:4, and the second is the color mode / space model conversion process from YUV4:4:4 to RGB4:4:4. According to the chrominance format information of the base layer image (preset or transmitted in metadata) and the chrominance format information of the combined processing process of the base layer image and the enhanced image (preset or transmitted in metadata), the chrominance format conversion is performed on the base layer image to obtain a base layer image of a new format. The conversion process can be completed according to a preset conversion method or according to a conversion method transmitted through metadata. The process can include a chrominance upsampling process and other transformation processes.

[0575] c) Other relevant transformations of the base layer image.

[0576] It should be noted that the present invention does not limit the order of the color mode / space model conversion process and the chromaticity format conversion process. The present invention does not limit the color mode / space model conversion process and the chromaticity format conversion process to be performed simultaneously, and only one or all of them may be performed.

[0577] It should be noted that before or after the color mode / space model conversion process and the chromaticity format conversion process described, there may be a corresponding mapping process described according to the preset or metadata. The mapping is not mandatory in this patent, and may be the following examples: normalization and inverse normalization, adding offset and reducing offset, multiplying by numerical coefficient and dividing by numerical coefficient, etc.

[0578] It should be noted that before or after the color mode / space model conversion process and the chroma format conversion process described, there may be a corresponding up / down sampling process described according to the preset or metadata. After sampling, a basic image with a new resolution can be obtained.

[0579] It should be noted that the metadata may include processing information for the base layer and the enhancement layer, or preset processing information may be used. The base layer Base may be processed based on the metadata to obtain BaseAfter. The processing scheme may be:

[0580] 1.BaseAfter[i]=Base[i].

[0581] 2. The metadata contains the first preset value THH and the second preset value THL of the basic layer data, using BaseAfter[i]=Base[i]*THH+(A-Base[i])*THL, where A is the maximum value stored in Base, and when Base is normalized to between 0 and 1.0, A is 1.0.

[0582] 3. The metadata includes a first preset value THH and a second preset value THL of the basic layer data, where the basic layer data can be non-linear coded data such as PQ / HLG / LOG / Gamma. First, Base[i] is converted to the linear domain, and then BaseAfter[i]=Base[i]*THH+(A-Base[i])*THL is used, where A is the maximum value stored in Base. When Base is normalized to between 0 and 1.0, A is 1.0.

[0583] 4. The metadata contains the preset value THL of the basic layer data, using BaseAfter[i]=Base[i]+THL.

[0584] 5. The metadata contains the preset value THH of the base layer data, using BaseAfter[i]=THH+A-Base[i], where A is the maximum value stored in Base, and when Base is normalized to between 0 and 1.0, A is 1.0.

[0585] 6. The metadata contains the parameters of the mapping relationship, and TME() is obtained according to the parameters, BaseAfter[i]=TMB(Base[i]).

[0586] 7. The metadata contains the parameters of the mapping relationship, and TME() is obtained according to the parameters, BaseAfter[i]=TME()*Base[i].

[0587] The mapping relationship has various forms, such as Sigmoid, cubic spline, gamma, straight line or piecewise curve, etc. or their inverse function forms, which are not limited in the present invention. The following curve represented by formula (19) or its inverse curve can be adopted:

[0588]

[0589] It should be noted that L and L' can be normalized optical signals or electrical signals, which is not limited in the present invention. It can also be a transformation in the form of a neural network: obtaining the type information of the neural network, such as transformer, convolutional neural network, etc.; constructing a neural network according to the network information in the enhanced data; and processing the enhanced data using a neural network. It should be noted that it can be global tone mapping or local tone mapping, which is not limited in the present invention.

[0590] 3. Acquisition of enhancement layer image.

[0591] The enhanced layer image serves as a bridge between the base layer and the final merged image. Any format can be used. The corresponding transfer function can be used during encoding and transmission. The decoding end decodes according to the preset information or the relevant information in the metadata to obtain the enhanced layer image. This process includes decoding the received code stream through any decoder (HEVC, JPEG, ProRes, HEIF, AV1F, etc.) to obtain the image data. The image data obtained in this process includes image data in any color space mode (RGB, YUV, YCbCr, CIELAB, HSV, HSL, ICtCp, etc.). It should be noted that the present invention does not limit the code stream format. In terms of color space, it can be YUV or RGB. In terms of the bit width of the data, it can be 8 bits, 10 bits, or 12 bits, etc. In terms of the numerical domain, it can be a linear domain or a non-linear domain such as PQ / HLG / Gamma / log.

[0592] a) The first color mode / space conversion method of enhancing the image.

[0593] If the base layer image obtained after decoding is in the first color space (such as RGB), the enhancement layer image is in the second color space (such as YUV), and the new dynamic range image after synthesis needs to be in the first color space, then the enhancement layer image needs to be converted to the first color space first. The transformation method includes the default matrix method or passing through metadata (such as 3x3 transformation matrix).

[0594] (No transformation) If the base layer image obtained after decoding is in the first color space (such as RGB), the enhancement layer image is also in the first color space (RGB), and the new dynamic range image after synthesis is required to be in the first color space, no transformation is performed or the default identity transformation is used.

[0595] If the base layer image obtained after decoding is in the first color space and the first color space has chroma and luma separation (such as YUV, CIELAB, etc.), the enhancement layer image is also in the first color space (YUV) and the new dynamic range image after synthesis is required to be in the second color space (RGB), then the enhancement layer image needs to be converted to the first color space first. The conversion method includes the default conversion method or the conversion method transmitted through metadata.

[0596] If the base layer image obtained after decoding is in the first color space (such as HSV), the enhancement layer image is in the second color space (such as YUV), and the new dynamic range image after synthesis is required to be in the third color space (such as RGB), it is necessary to perform a preset transformation when synthesizing the new dynamic range image or perform a color space transformation according to metadata.

[0597] b) A second color space / mode conversion method for enhancing the image.

[0598] If the base layer image obtained after decoding is in the first color space (such as RGB), and the base layer and enhanced layer image / enhancement data combination processing process is in the second color space (such as YUV), the enhanced layer image needs to be converted to the second color space first. According to the enhanced layer image / enhancement data color space model information (preset or transmitted in metadata) and the color space model information of the base layer and enhanced layer image combination processing process (preset or transmitted in metadata), the enhanced layer image / enhancement data is transformed / converted to obtain a new format of enhanced layer image / enhancement data. The transformation method includes a default method or a description of the conversion process transmitted in metadata, which can be a 3x3 matrix method specified by ITU.

[0599] (No transformation) If the base layer image obtained after decoding is in the first color space (such as RGB), and the base layer and enhanced image combined processing process is in the first color space, no transformation is performed or the default identity transformation is used.

[0600] c) Enhanced image chroma format conversion.

[0601] If the luminance component and chrominance component formats of the enhanced layer image are in the first chrominance format (such as YUV4:2:0), the luminance component and chrominance component formats of the new enhanced layer image to be processed and converted are in the second chrominance format (such as YUV4:4:4), or the chrominance format information of the combined processing process of the base layer and the enhanced image is in the third chrominance format (such as RGB4:4:4), then the enhanced image needs to be converted during the conversion process. This includes: the first is the upsampling conversion from YUV4:2:0 to YUV4:4:4, and the second is the color space model conversion from YUV4:4:4 to RGB4:4:4. According to the chrominance format information of the enhanced layer image (preset or transmitted in metadata) and the chrominance format information of the combined processing process of the base layer and the enhanced image (preset or transmitted in metadata), the chrominance format conversion is performed on the enhanced image to obtain a base layer image of a new format. The conversion process can be completed according to a preset conversion method or according to a conversion method transmitted through metadata.

[0602] d) Other relevant transformations of enhancement layer images / enhancement data.

[0603] It should be noted that the present invention does not limit the order of the color mode / space model conversion process and the chromaticity format conversion process. The present invention does not limit the color mode / space model conversion process and the chromaticity format conversion process to be performed simultaneously, and only one or all of them may be performed.

[0604] It should be noted that before and after the color mode / space model conversion process and the chroma format conversion process described, there can be a corresponding mapping process described according to the preset or metadata. The mapping is not mandatory in this patent, and can be the following examples: normalization and inverse normalization, adding offset and reducing offset, multiplying and dividing numerical coefficients, upsampling and downsampling of images, etc.

[0605] It should be noted that before or after the color mode / space model conversion process and the chroma format conversion process described, there may be a corresponding up / down sampling process described according to the preset or metadata. After sampling, an enhanced image with a new resolution can be obtained.

[0606] It should be noted that the metadata may include processing information for the base layer and the enhancement layer, or preset processing information may be used. Taking the enhancement layer as an example, EnhanceAfter may be obtained by processing the enhancement layer Enhance based on the metadata. The processing scheme may be:

[0607] 1.EnhanceAfter[i]=Enhance[i].

[0608] 2. The metadata contains the first preset value THH and the second preset value THL of the enhancement layer data, using EnhanceAfter[i]=Enhance[i]*THH+(A-Enhance[i])*THL, where A is the maximum value stored in Enhance, and when Enhance is normalized to between 0 and 1.0, A is 1.0.

[0609] 3. The metadata includes a first preset value THH and a second preset value THL of the enhancement layer data, where the enhancement layer data can be non-linear coding data such as PQ / HLG / LOG / Gamma. First, convert Enhance[i] to the linear domain, and then use EnhanceAfter[i]=Enhance[i]*THH+(A-Enhance[i])*THL, where A is the maximum value stored in Enhance. When Enhance is normalized to between 0 and 1.0, A is 1.0.

[0610] 4. The metadata contains the preset value THL of the enhancement layer data, using EnhanceAfter[i]=Enhance[i]+THL.

[0611] 5. The metadata contains the preset value THH of the enhancement layer data, using EnhanceAfter[i]=THH+A-Enhance[i], where A is the maximum value stored in Enhance, and when Enhance is normalized to between 0 and 1.0, A is 1.0.

[0612] 6. The metadata contains the parameters of the mapping relationship, and TME() is obtained according to the parameters, EnhanceAfter[i]=TMB(Enhance[i]).

[0613] 7. The metadata contains the parameters of the mapping relationship, and TME() is obtained according to the parameters, EnhanceAfter[i]=TME()*Enhance[i].

[0614] The mapping relationship has various forms, such as Sigmoid, cubic spline, gamma, straight line or piecewise curve, etc. or their inverse function forms, which are not limited in the present invention. The curve represented by formula (19) above or its inverse curve can be adopted, and formula (19) can be expressed again as follows:

[0615]

[0616] It should be noted that L and L' can be normalized optical signals or electrical signals, which is not limited in the present invention. It can also be a transformation in the form of a neural network: obtaining the type information of the neural network, such as transformer, convolutional neural network, etc.; constructing a neural network according to the network information in the enhanced data; and processing the enhanced data using a neural network. It should be noted that it can be global tone mapping or local tone mapping, which is not limited in the present invention.

[0617] Second, the base layer image and the enhancement layer image are merged according to preset information or metadata to obtain a reconstructed new dynamic range image.

[0618] According to the previous process, the enhanced layer processing result EnhanceAfter can be obtained, and combined with BaseAfter, the merged result newPictue can be obtained.

[0619] a) A method for obtaining the merging of multiple color channels.

[0620] According to the information of "how to combine the color mode multi-channel of the base layer with the color mode multi-channel or single channel of the enhanced data" transmitted or preset in the metadata, the multi-color channel merging method of the base layer and the enhanced layer image / enhanced data is obtained. The base layer image and the enhanced layer image / enhanced data are both in a certain color mode or color space, and different color modes or color spaces contain different numbers of color channels. For example, the YUV color mode contains three color channels of Y, U, and V, the grayscale color mode contains only one color channel, and the CMYK color mode contains four color channels.

[0621] Assuming that BaseAfter is a single color channel and EnhanceAfter is a single color channel, the merging method information may include a one-to-one combination method according to preset merging method information or merging method information obtained from metadata.

[0622] Assuming that BaseAfter is a multi-color channel and EnhanceAfter is a single color channel, the merging method information may include one-to-one or many-to-one combination according to the preset merging method information or the merging method information obtained from metadata. For example:

[0623] Many-to-one: For example, if BaseAfter is a three-channel image (such as RGB) with inseparable brightness and chrominance, and EnhanceAfter is a single-channel C image, then perform R*C, G*C, B*C or R+C, G+C, B+C, that is, multiply or add each channel separately.

[0624] One-to-one: For example, if BaseAfter is a three-channel image with separated brightness and chrominance (such as YUV, Lab, HSV, etc.), then perform Y*C, U*1, V*1 or Y+C, U+0, V+0, that is, only multiply or add the brightness channel, but not the chrominance channel.

[0625] Assuming that BaseAfter has multiple channels and EnhanceAfter has multiple channels, the merging mode information may include a many-to-many combination mode according to preset merging mode information or merging mode information obtained from metadata.

[0626] Three to two: For example, BaseAfter is a brightness and color separation model (such as Lab, YUV), and EnhanceAfter has two color channels (such as YC), then the merging operation is performed on the corresponding brightness and color channels, such as baseY*enhenceAfterY, baseU*enhenceAfterC, baseV*enhenceAfterC.

[0627] Three to two: For example, BaseAfter is a brightness and color separation model (such as HSV, HIS, etc.), and EnhanceAfter has two color channels (such as YC), then the merging operation is performed on the corresponding brightness (V or I channel, etc.) and color saturation or hue channel, such as baseV*enhenceAfterY, baseS*enhenceAfterC, baseH remains unchanged, or baseV*enhenceAfterY, baseH*enhenceAfterC, baseS remains unchanged.

[0628] Many-to-many: For example, if BaseAfter and EnhanceAfter have the same color space model (such as RGB, YUV), the merge operation is performed on the corresponding channels, such as R*R, G*G, B*B.

[0629] Many-to-many: For example, if the color space model of BaseAfter is RGB, and the color space model of EnhanceAfter is a color space model with separated brightness and chromaticity (such as YUV, Lab), it is necessary to first convert from the second color space to the first color space (such as YUV to RGB), and then merge the corresponding channels.

[0630] Many-to-many: For example, if the base layer and the enhancement layer have multiple channels and the number of channels is greater than three (such as RGBA), a reasonable two-layer image merging operation should be performed based on the characteristics of the channels and information such as brightness and chromaticity separation. It can be done by processing only the common color channels such as RGB on the corresponding channels.

[0631] b) Perform merging calculation according to the method of merging multiple color channels.

[0632] Solution 1: newPictue[i] = BaseAfter[i]*f(EnhanceAfter[i]), where f(x) is the numeric domain conversion function.

[0633] Solution 2: newPictue[i]=BaseAfter[i]+f(EnhanceAfter[i]), where f(x) is the numerical domain conversion function.

[0634] Solution 3: newPictue[i]=g(BaseAfter[i]*f(EnhanceAfter[i])), where f(x) is the numerical domain conversion function and g(x) is the mapping function.

[0635] Solution 4: newPictue[i]=g(BaseAfter[i]+f(EnhanceAfter[i])), where f(x) is the numerical domain conversion function and g(x) is the mapping function.

[0636] It should be noted that the present invention does not limit the form of f(x) and g(x), that is, it does not limit BaseAfter[i] and EnhanceAfter[i] to any domain, which can be a linear domain or a PQ domain, a log domain, etc. In addition, the present invention does not limit whether other image processing is performed on the reconstructed new dynamic range image newPictue after obtaining it and before displaying it.

[0637] The present application also provides another embodiment, wherein the base layer is in the first format (first transfer function), an image in the second format (second transfer function) is obtained through the enhancement layer, and the enhancement layer is the second or third transfer function; or

[0638] The base layer is in the first color space, and the image in the second color space is obtained through the enhancement layer; or

[0639] The base layer is in the first color gamut space, which is changed to the second color gamut space through the 3x3 matrix transmitted in the metadata or by default, and then a higher dynamic range image of the second color gamut space is obtained through the enhancement layer; or

[0640] The basic layer is the first color space YUV, the enhanced layer is the first YUV or the second RGB color space, and after the basic layer is converted to the second RGB color space, an image in the second color space is obtained through the enhanced layer.

[0641] In the embodiments of the present application, the symbol * and the symbol × can be interchanged.

[0642] The technical solution of the present application can use a mapping function to process a base image during a transformation process of the base image according to metadata; can use a mapping function to process enhanced data during a transformation process of the enhanced data according to metadata; can perform up and down sampling during a transformation process of the enhanced data according to metadata, use an interpolation algorithm in a texture sampling algorithm in OPENGL, Vulkan, or Metal for processing, or use an index to select a better algorithm from a set of interpolation algorithms.

[0643] The above describes the data processing method provided by the present application in conjunction with the accompanying drawings. The following describes an encoding device provided by the present application in conjunction with the accompanying drawings. Fig.19 The schematic diagram of the structure of a coding device provided in the present application is as follows. The coding device 1900 includes: a processing module 1910 and a coding module 1920. The processing module 1910 is used to execute step S1410, and the coding module 1920 is used to execute step S1420. In some possible examples, the coding device 1900 may also include a transceiver module 1930, which is used to execute step S1430. The coding device can be used to implement Fig.10 Functions implemented by the source device 1010.

[0644] It should be understood that the encoder of the aforementioned embodiment may correspond to the encoding device 1900, and may correspond to the corresponding subject executing the method embodiment of the present application, and the operations and / or functions of each module in the encoding device 1900 are respectively for implementing the corresponding processes of each method of the embodiment, which will not be repeated here for the sake of brevity.

[0645] The above describes the encoding device provided by the present application in conjunction with the accompanying drawings, and the following describes the encoding device provided by the present application in conjunction with the accompanying drawings. Fig. 20 A schematic diagram of the structure of an encoding device provided in this application, such as Fig. 20 As shown, the encoding device 2000 may include a processor 2010 for executing a program or instruction stored in a memory 2020. When the program or instruction stored in the memory 2020 is executed, the processor is used to execute the method in the aforementioned embodiment of the present application. The encoding device 2000 may be used to implement the above Fig.10 Functions implemented by the source device 1010.

[0646] In a possible scenario, the encoding device 2000 may further include a communication interface 2030. Fig. 20 The communication interface 2030 is indicated by a dotted line as being optional for the encoding device 2000. When the communication interface 2030 implements the function of the transceiver module 1910, it is used to receive the first image.

[0647] Among them, the number of processors 2010, memories 2020 and communication interfaces 2030 does not constitute a limitation on the embodiments of the present application, and during specific implementation, they can be arbitrarily configured according to business requirements.

[0648] In one possible scenario, the memory 2020 is located outside the encoding device 2000 .

[0649] In one possible scenario, the encoding device 2000 includes a memory 2020, the memory 2020 is connected to at least one processor 2010, and the memory 2020 stores instructions that can be executed by the at least one processor 2010. The memory 2020 can also be used to store the first image, the base layer data of the first image, the enhancement layer data of the first image, and the metadata of the first image. Fig. 20 The memory 2020 is indicated by a dotted line as being optional for the encoding device 2000 .

[0650] The processor 2010 and the memory 2020 may be coupled via an interface circuit or may be integrated together, which is not limited here. The processor 2010 is used to implement the above Fig.19 The functions of the processing module 1910 in the described encoding device.

[0651] The specific connection medium between the processor 2010, the memory 2020 and the communication interface 2030 is not limited in the embodiment of the present application. Fig. 20 The processor 2010, the memory 2020 and the communication interface 2030 are connected via a bus 2040. Fig. 20The connections between other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 20 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0652] The encoding device provided by the present application is described above in conjunction with the accompanying drawings, and the decoding device provided by the present application is described below in conjunction with the accompanying drawings. Fig.21 A schematic diagram of the structure of a decoding device provided in this application, such as Fig.21 As shown, the decoding device includes: a transceiver module 2110, a processing module 2120 and a conversion module 2130. The transceiver module 2110 is used to execute step S1440, the processing module 2120 is used to execute step S1450, and the conversion module 2130 is used to execute step S1460.

[0653] The processing module 2120 is used to receive a code stream. The processing module 2120 is used to: determine the target data according to the conversion information carried in the metadata. The metadata is metadata of the first image. The target data includes the base layer data of the first image and / or the enhancement layer data of the first image, or the target data includes the second image. The second image is obtained based on the base layer data and the enhancement layer data. The conversion module 2130 is used to: convert the target data according to the conversion information.

[0654] It should be understood that the decoder of the aforementioned embodiment may correspond to the decoding device 2100, and may correspond to the corresponding subject executing the method embodiment of the present application, and the operations and / or functions of each module in the decoding device 2100 are respectively for implementing the corresponding processes of each method of the embodiment, and for the sake of brevity, they will not be repeated here.

[0655] The above describes the decoding device provided by the present application in conjunction with the accompanying drawings, and the following describes the decoding device provided by the present application in conjunction with the accompanying drawings. Fig. 22 A schematic diagram of the structure of a decoding device provided in this application, such as Fig. 22 As shown, the decoding device 2200 includes: a processor 2210 and a memory 2220. The processor 2210 is used to execute the program or instruction stored in the memory 2220. When the program or instruction stored in the memory 2220 is executed, the processor 2210 is used to execute Fig.14 The method in the embodiment.

[0656] In a possible scenario, the decoding device 2200 may further include a communication interface 2230. Fig. 22 The communication interface 2230 is indicated by a dashed line as being optional for the decoding device 2200. The communication interface 2230 is used to implement the above Fig.21The processor 2210 is used to implement the functions of the transceiver module 2110 in the above method embodiment to execute S1440. Fig.21 The functions of the processing module 2120 and the conversion module 2130 are used to execute S1450 and S1460 in the method embodiment.

[0657] Among them, the number of processors 2210, memories 2220 and communication interfaces 2230 does not constitute a limitation on the embodiments of the present application. During specific implementation, they can be arbitrarily configured according to business requirements.

[0658] In one possible scenario, the memory 2220 is located outside the decoding device 2200 .

[0659] In a possible scenario, the decoding device 2200 includes a memory 2220, the memory 2220 is connected to at least one processor 2210, and the memory 2220 stores instructions that can be executed by at least one processor 2210. The memory 2220 is used to store the code stream, and the basic layer data of the first image obtained by decoding, the enhancement layer data, and the metadata of the first image, etc. Fig. 22 The memory 2220 is indicated by a dotted line as being optional for the decoding device 2200 .

[0660] The processor 2210 and the memory 2220 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0661] The specific connection medium between the processor 2210, the memory 2220 and the communication interface 2230 is not limited in the embodiment of the present application. Fig. 22 The processor 2210, the memory 2220 and the communication interface 2230 are connected via a bus 2240. Fig. 22 The connections between other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 22 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0662] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor implemented by reading software code stored in a memory.

[0663] Exemplarily, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0664] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (directrambus RAM, DR RAM).

[0665] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0666] It should be noted that the memory described herein is intended to include, without being limited to, these and any other suitable types of memory.

[0667] An embodiment of the present application further provides a computer-readable storage medium, including a program or an instruction. When the program or the instruction is executed on a computer, the encoding and decoding method in the method embodiment described above is implemented.

[0668] The term "at least one" in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where any of a, b, c, ab, ac, bc, or abc can include a single a, a single b, a single c, and can also include multiple a, multiple b, and multiple c.

[0669] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria, and do not indicate the difference in the content, priority or importance of the two criteria.

[0670] In addition, the terms "including" and "having" in the embodiments, claims and drawings of the present application are not exclusive. For example, a process, method, system, product or device including a series of steps or modules / units is not limited to the listed steps or modules, and may also include steps or modules / units that are not listed.

[0671] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0672] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0673] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0674] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0675] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the target data according to the conversion information carried in the metadata; wherein the metadata is metadata of a first image, and the target data includes base layer data of the first image and / or enhancement layer data of the first image; or, the target data includes a second image, wherein the second image is obtained according to the base layer data and the enhancement layer data; The target data is converted according to the conversion information.

2. The method according to claim 1, characterized in that: The metadata is carried in the bitstream. The method further includes: acquiring the metadata from the code stream.

3. The method according to claim 1 or 2, characterized in that: The conversion information includes conversion position information and conversion scheme information, wherein the conversion position information is used to indicate the position where the conversion occurs, and the conversion scheme information is used to indicate the conversion scheme used when converting the target data. The conversion occurs before or after obtaining the second image.

4. The method according to any one of claims 1 to 3, characterized in that In the case where the conversion occurs before the second image is obtained, the conversion information further includes conversion object information, where the conversion object information is used to indicate a conversion object on which the conversion scheme needs to be executed, and the conversion object includes: Base layer data of the first image; or enhancement layer data of the first image; or, Base layer data of the first image and enhancement layer data of the first image.

5. The method according to claim 4, characterized in that In the case where the conversion object includes the basic layer data, the conversion scheme information includes: a basic layer data conversion scheme, where the basic layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the basic layer data: Color gamut conversion; Chroma format conversion; Color compensation; or Color mode conversion.

6. The method according to claim 5, characterized in that The basic layer data conversion scheme is also used to indicate conversion parameters used when converting the basic layer data.

7. The method according to claim 6, characterized in that The conversion parameters include: compensation matrix; or, a transformation matrix; or, a conversion function; or, Compensation value.

8. The method according to claim 4, characterized in that In the case where the conversion object includes the enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, where the enhancement layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the enhancement layer data: Color mode conversion; Chroma format conversion; or, Channel conversion, where the channel conversion is used to restore the channels of the enhancement layer data to the original channels.

9. The method according to claim 8, characterized in that The enhancement layer data conversion scheme is also used to indicate conversion parameters used when converting the enhancement layer data.

10. The method according to claim 9, characterized in that: The conversion parameters include: a transformation matrix; or, a conversion function; or, Channel conversion function.

11. The method according to any one of claims 1 to 10, characterized in that The conversion information includes channel merging information; the channel merging information is used to indicate: a combination scheme of the channel of the basic layer data and the channel of the enhancement layer data.

12. The method according to claim 11, characterized in that The color mode of at least one of the base layer data and the enhancement layer data is a red, green, blue (RGB) mode or a grayscale mode; Alternatively, the color mode of the base layer data is an RGB mode, and the color mode of the enhancement layer data is any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode; Alternatively, the color mode of the base layer data is any one of a YUV mode, a Lab mode, a HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode, and the color mode of the enhancement layer data is an RGB mode; Alternatively, the color mode of the basic layer data and the color mode of the enhanced layer data are any one of: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode.

13. The method according to claim 12, characterized in that In the case where the color modes of the basic layer data and the enhancement layer data are both RGB modes, the channel merging information is used to indicate: an R channel of the basic layer data is merged with an R channel of the enhancement layer data, a G channel of the basic layer data is merged with a G channel of the enhancement layer data, and a B channel of the basic layer data is merged with a B channel of the enhancement layer data; When the color mode of the base layer data is the RGB mode and the color mode of the enhancement layer data is the grayscale mode, the channel merging information is used to indicate: the R channel of the base layer data is merged with the grayscale value of the enhancement layer data, the G channel of the base layer data is merged with the grayscale value of the enhancement layer data, and the B channel of the base layer data is merged with the grayscale value of the enhancement layer data; When the color mode of the base layer data is a grayscale mode and the color mode of the enhancement layer data is an RGB mode, the channel merging information is used to indicate: an R channel of the enhancement layer data is merged with a grayscale value of the base layer data, a G channel of the enhancement layer data is merged with a grayscale value of the base layer data, and a B channel of the enhancement layer data is merged with a grayscale value of the base layer data; In the case where the color modes of the base layer data and the enhancement layer data are both grayscale modes, the channel merging information is used to indicate that the grayscale value of the base layer data and the grayscale value of the enhancement layer data are merged; In the case where the color mode of the base layer data and the color mode of the enhancement layer data are any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode, the channel merging information is used to indicate that the brightness channel of the base layer data and the brightness channel of the enhancement layer data are merged; In the case where the color modes of the base layer data and the enhancement layer data are both any one of the YUV mode, the Lab mode, the HSV mode, the YCbCr mode, and the ICtCp mode, the channel merging information is used to indicate that the brightness channel of the base layer data and the brightness channel of the enhancement layer data are merged; When the color mode of the base layer data is a grayscale mode and the color mode of the enhancement layer data is any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, and an ICtCp mode, the channel merging information is used to indicate that a channel of the base layer data and a brightness channel of the enhancement layer data are merged; When the color mode of the basic layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, and the color mode of the enhanced layer data is a grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the channel of the enhanced layer data are merged.

14. The method according to claim 5 or 8, characterized in that: The color mode conversion includes: Convert the color mode of the basic layer data so that the converted color mode of the basic layer data is the same as the color mode of the enhancement layer data, or convert the color mode of the enhancement layer data so that the converted color mode of the enhancement layer data is the same as the color mode of the basic layer data.

15. The method according to any one of claims 1 to 3, characterized in that: In the case where the conversion occurs after the second image is obtained, the conversion scheme information includes: Color gamut conversion; or, Chroma format conversion; or, Color compensation; or Color mode conversion.

16. The method according to claim 15, characterized in that The conversion scheme information also includes conversion parameters; The conversion parameters include: compensation matrix; or, a transformation matrix; or, a conversion function; or, Compensation value.

17. A coding method, characterized in that: Determine, according to the acquired first image, base layer data of the first image, enhancement layer data of the first image, and metadata of the first image, wherein the metadata carries conversion information; The base layer data, the enhancement layer data and the metadata are encoded to obtain a code stream.

18. The method according to claim 17, characterized in that The conversion information is conversion indication information, and the conversion indication information includes conversion position information and conversion scheme information, the conversion position information is used to indicate the position where the conversion scheme occurs, and the conversion scheme information is used to indicate the conversion scheme. The conversion occurs before obtaining the second image or after obtaining the second image, and the second image indicates a new image synthesized based on the received code stream.

19. The method according to claim 17 or 18, characterized in that In the case where the conversion occurs before the second image is obtained, the conversion information further includes conversion object information, where the conversion object information is used to indicate a conversion object on which the conversion scheme needs to be executed, and the conversion object includes: Base layer data; or, enhancement layer data; or, The base layer data and the enhancement layer data.

20. The method according to claim 19, characterized in that In the case where the conversion object includes the basic layer data, the conversion scheme information includes: a basic layer data conversion scheme, where the basic layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the basic layer data: Color gamut conversion; Chroma format conversion; Color compensation; or Color mode conversion.

21. The method according to claim 20, characterized in that The basic layer data conversion scheme is also used to indicate conversion parameters used when converting the basic layer data.

22. The method according to claim 21, characterized in that The conversion parameters include: compensation matrix; or, a transformation matrix; or, a conversion function; or, Compensation value.

23. The method according to claim 19, characterized in that In the case where the conversion object includes the enhancement layer data, the conversion scheme information includes: an enhancement layer data conversion scheme, where the enhancement layer data conversion scheme is used to indicate that at least one of the following conversions is performed on the enhancement layer data: Color mode conversion; Chroma format conversion; or, Channel conversion, where the channel conversion is used to restore the channels of the enhancement layer data to the original channels.

24. The method according to claim 23, characterized in that The enhancement layer data conversion scheme is also used to indicate conversion parameters used when converting the enhancement layer data.

25. The method according to claim 24, characterized in that: The conversion parameters include: a transformation matrix; or, a conversion function; or, Channel conversion function.

26. The method according to any one of claims 18 to 25, characterized in that The conversion information includes channel merging information; the channel merging information is used to indicate: a combination scheme of the channel corresponding to the color mode of the basic layer data and the channel corresponding to the color mode of the enhancement layer data.

27. The method according to claim 26, characterized in that The color mode of at least one of the base layer data and the enhancement layer data is a red, green, blue (RGB) mode or a grayscale mode; Alternatively, the color mode of the base layer data is an RGB mode, and the color mode of the enhancement layer data is any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode; Alternatively, the color mode of the base layer data is any one of a YUV mode, a Lab mode, a HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode, and the color mode of the enhancement layer data is an RGB mode; Alternatively, the color mode of the basic layer data and the color mode of the enhanced layer data are any one of: YUV mode, Lab mode, HSV mode, YCbCr mode, ICtCp mode, and grayscale mode.

28. The method according to claim 27, characterized in that In the case where the color modes of the basic layer data and the enhancement layer data are both RGB modes, the channel merging information is used to indicate: an R channel of the basic layer data is merged with an R channel of the enhancement layer data, a G channel of the basic layer data is merged with a G channel of the enhancement layer data, and a B channel of the basic layer data is merged with a B channel of the enhancement layer data; When the color mode of the base layer data is the RGB mode and the color mode of the enhancement layer data is the grayscale mode, the channel merging information is used to indicate: the R channel of the base layer data is merged with the grayscale value of the enhancement layer data, the G channel of the base layer data is merged with the grayscale value of the enhancement layer data, and the B channel of the base layer data is merged with the grayscale value of the enhancement layer data; When the color mode of the base layer data is a grayscale mode and the color mode of the enhancement layer data is an RGB mode, the channel merging information is used to indicate: an R channel of the enhancement layer data is merged with a grayscale value of the base layer data, a G channel of the enhancement layer data is merged with a grayscale value of the base layer data, and a B channel of the enhancement layer data is merged with a grayscale value of the base layer data; In the case where the color modes of the base layer data and the enhancement layer data are both grayscale modes, the channel merging information is used to indicate that the grayscale value of the base layer data and the grayscale value of the enhancement layer data are merged; In the case where the color mode of the base layer data and the color mode of the enhancement layer data are any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, an ICtCp mode, and a grayscale mode, the channel merging information is used to indicate that the brightness channel of the base layer data and the brightness channel of the enhancement layer data are merged; In the case where the color modes of the base layer data and the enhancement layer data are both any one of the YUV mode, the Lab mode, the HSV mode, the YCbCr mode, and the ICtCp mode, the channel merging information is used to indicate that the brightness channel of the base layer data and the brightness channel of the enhancement layer data are merged; When the color mode of the base layer data is a grayscale mode and the color mode of the enhancement layer data is any one of a YUV mode, a Lab mode, an HSV mode, a YCbCr mode, and an ICtCp mode, the channel merging information is used to indicate that a channel of the base layer data and a brightness channel of the enhancement layer data are merged; When the color mode of the basic layer data is any one of the YUV mode, Lab mode, HSV mode, YCbCr mode, and ICtCp mode, and the color mode of the enhanced layer data is a grayscale mode, the channel merging information is used to indicate: the brightness channel of the basic layer data and the channel of the enhanced layer data are merged.

29. The method according to claim 20 or 23, characterized in that The color mode conversion includes: Convert the color mode of the basic layer data so that the converted color mode of the basic layer data is the same as the color mode of the enhancement layer data, or convert the color mode of the enhancement layer data so that the converted color mode of the enhancement layer data is the same as the color mode of the basic layer data.

30. The method according to claim 17 or 18, characterized in that In the case where the conversion occurs after the second image is obtained, the conversion scheme information includes: Color gamut conversion; or, Chroma format conversion; or, Color compensation; or Color mode conversion.

31. The method according to claim 30, characterized in that The conversion scheme information also includes conversion parameters; The conversion parameters include: compensation matrix; or, a transformation matrix; or, a conversion function; or, Compensation value.

32. The method according to any one of claims 1 to 31, characterized in that The conversion information includes at least one or more of the following: Color mode information of the base layer data of the first image; Color mode information of the enhancement layer data of the first image; and / or, The color mode information corresponding to the basic layer data of the first image and the enhanced layer data of the first image needs to be transformed and processed.

33. The method according to claim 32, characterized in that When the image color space model of the basic layer data is a first color mode, the color space model when the basic layer data and the enhanced layer data are processed in combination is a second color mode, and the first color mode is different from the second color mode, the transformed and processed color mode information includes a color mode conversion method.

34. The method according to claim 33, characterized in that The first color mode and the second color mode are one of the following: RGB, YUV, YCbCr, Lab, HSV, ICtCp.

35. The method according to any one of claims 1 to 34, characterized in that The conversion information includes at least one or more of the following: Chroma format information of base layer data of the first image; Chroma format information of enhancement layer data of the first image; and / or, The chroma format information corresponding to the base layer data of the first image and the enhancement layer data of the first image that needs to be transformed and processed.

36. The method according to claim 35, characterized in that In the case where the chroma format information of the base layer data of the first image is the first chroma format information, the chroma format information when the base layer data is processed in combination with the enhancement layer data is the second chroma format information, and the first chroma format information is different from the second chroma format information, The chroma format information that needs to be transformed and processed includes chroma format conversion information.

37. The method according to claim 36, characterized in that The first chroma format information and the second chroma format information are one of the following: RGB, grayscale, YUV4:4:4, YUV4:2:2, YUV4:2:

0.

38. The method according to any one of claims 1 to 37, characterized in that: The first image and the second image are HDR or SDR images.

39. A chip, characterized in that: include: Processor and power supply circuit; The power supply circuit is used to supply power to the processor; The processor is used to execute the method of any one of claims 1 to 16 or any one of claims 32 to 38; and / or, the processor is used to execute the method of any one of claims 17 to 38.

40. A decoding device, characterized in that: The decoding device comprises: A processing module, determining target data according to conversion information carried in metadata; wherein the metadata is metadata of a first image, and the target data includes base layer data of the first image and / or enhancement layer data of the first image; or, the target data includes a second image; and the second image is obtained according to the base layer data and the enhancement layer data; A conversion module converts the target data according to the conversion information.

41. A coding device, characterized in that The encoding device comprises: A processing module, which determines, according to the acquired first image, base layer data of the first image, enhancement layer data of the first image, and metadata of the first image, wherein the metadata carries conversion information; The encoding module encodes the base layer data, the enhancement layer data and the metadata to obtain a code stream.

42. A codec, characterized in that: It comprises a memory and a processor, the memory is used to store computer instructions; when the processor executes the computer instructions, it implements any one of claims 1 to 16 or any one of claims 32 to 38; and / or when the processor executes the computer instructions, it implements any one of claims 17 to 38.

43. A coding and decoding system, characterized in that: Includes encoder and decoder; The encoder is used to obtain, according to the first image, basic layer data, enhancement layer data and metadata carrying conversion information of the first image, and encode the basic layer data, the enhancement layer data and the metadata to obtain a code stream of the first image, thereby implementing the method according to any one of claims 17 to 38; The decoder is used to determine and convert target data according to conversion information carried by metadata of the first image, thereby implementing the method of any one of claims 1 to 16 or any one of claims 32 to 38.

44. A computer-readable storage medium, characterized in that The storage medium stores a computer program or instruction, which, when executed by a processing device, implements the method of any one of claims 1 to 16 or any one of claims 32 to 38; and / or, when executed by a processing device, implements the method of any one of claims 17 to 38.

45. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed on a processed device, it implements the method of any one of claims 1 to 16 or any one of claims 32 to 38; and / or, when the computer program or instruction is executed on a processed device, it implements the method of any one of claims 17 to 38.

Images