Metadata encoding and decoding method for multimedia

Abstract

The embodiment of the invention provides a multimedia metadata encoding and decoding method, and the encoding method comprises the steps: firstly, obtaining multimedia metadata; wherein the metadata of the multimedia comprises first data, and the first data is floating point type data; then, writing the second data into the code stream; wherein the second data is represented by the first data in a floating point data format. Therefore, the floating-point number finally obtained by the decoding end can be ensured to be consistent with the original floating-point number of the coding end, and precision loss is avoided.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of encoding and decoding, and particularly to a method for encoding and decoding metadata of multimedia. Background Art

[0002] As multimedia (such as audio, video, and images, etc.) develops from high definition to ultra high definition, the amount of multimedia data also increases accordingly, and thus the requirements for multimedia storage and transmission bandwidth are getting higher and higher; therefore, multimedia will be compressed (also known as encoded) before storage or transmission to reduce the memory occupied by storage and the transmission bandwidth.

[0003] Generally, while encoding multimedia, metadata of the multimedia is also encoded; afterwards, the encoded multimedia and the metadata of the multimedia can be encapsulated into a file (also known as a bitstream), and then the file (or bitstream) is stored or transmitted.

[0004] Currently, in some scenarios, metadata of multimedia (such as metadata of high dynamic range (HDR) images / videos) contains floating-point numbers. The solution of the prior art is that during the encoding process, after truncating the floating-point numbers to a certain length, they are encapsulated in a file or stream in the form of an American Standard Code for Information Interchange (ASCII) string; during the decoding process, the string is restored to a floating-point number again.

[0005] For example, truncate "0.131241241241……" to "0.1312412", and then represent the string "0.1312412" in the bitstream in the form of an Extensible Metadata Platform (XMP) or an Extensible Markup Language (XML). During the decoding process, change the string "0.1312412" back to the floating-point number 0.1312412. In this way, the precision of the floating-point numbers in the metadata will be lost, resulting in the floating-point numbers finally obtained at the decoding end being inconsistent with the original floating-point numbers at the encoding end. Summary of the Invention

[0006] In view of this, embodiments of the present application provide a method for encoding and decoding metadata of multimedia, which can ensure that the floating-point numbers finally obtained at the decoding end are consistent with the original floating-point numbers at the encoding end without precision loss.

[0007] It should be noted that the multimedia involved in the present application may include but are not limited to: images, videos, and audio.

[0008] Exemplarily, the present application can be applied to various image service (such as cloud photo album, etc.) scenarios, various video / audio service scenarios (such as live broadcast, on-demand, etc.), and so on. The embodiments of the present application are not limited thereto.

[0009] In a first aspect, an embodiment of the present application provides a method for encoding metadata of multimedia. The method includes: First, obtain the metadata of multimedia; wherein, the metadata of multimedia includes first data, and the first data is floating-point data; Subsequently, write second data into the bitstream; wherein, the second data is a representation form of the first data in a floating-point data format; After that,

[0010] Since for a specific syntax element, a codeword of a conventional length is usually transmitted in the bitstream, and since floating-point numbers are usually numbers with high precision, during the encapsulation process of syntax elements in existing standards, fixed-point conversion or conversion to other formats (such as ASCII strings, and length truncation is involved in the conversion process) will be performed, resulting in loss of precision. However, in the present application, the form stored in the computer memory as floating-point numbers in the bitstream (i.e., the floating-point data format of the second data) is used for transmission, which can ensure that the floating-point numbers stored in the memory at the sending end and the receiving end are exactly the same. In this way, the floating-point data obtained by the decoding end through the conversion of the second data is the same as the first data used for encoding at the encoding end, thus ensuring that the floating-point numbers finally obtained at the decoding end are the same as the original floating-point numbers at the encoding end.

[0011] Secondly, since the floating-point data in the metadata of multimedia at the encoding and decoding ends is the same, in scenarios where it is necessary to generate reconstructed multimedia based on the metadata of multimedia or display the reconstructed multimedia content according to the metadata of multimedia, the quality of the reconstructed multimedia or the visual experience when displaying the reconstructed multimedia content can be improved.

[0012] Exemplarily, the floating-point type includes 32-bit float type or 64-bit double type.

[0013] Exemplarily, the floating-point data format is a data format that stipulates each field constituting the floating-point data, the layout of each field, and the arithmetic interpretation.

[0014] For example, if the first data is "0.125490203", which is the mathematical expression of a floating-point number, the second data can be "0X3E008081", which is the binary expression of the floating-point number in the computer.

[0015] Exemplarily, the first data can be one or more, and the second data can be one or more; based on one first data, one second data can be generated; that is to say, multiple first data and multiple second data correspond one by one.

[0016] It should be understood that the metadata of the multimedia may further include other data in addition to the first data, and the methods of the prior art can be used to encode and decode the other data in the metadata of the multimedia, and the embodiments of the present application do not limit this; the embodiments of the present application illustrate the encoding and decoding process of the floating-point data in the multimedia data.

[0017] It should be understood that the metadata of the multimedia may be the metadata of an image, the metadata of an audio, the metadata of a video, etc., and the embodiments of the present application do not limit this.

[0018] Exemplarily, when the multimedia is an image or a video, the second data can be encapsulated according to the image encapsulation format to obtain a bitstream. Among them, the image encapsulation format includes but is not limited to: JPEG JFIF encapsulation format, JPEG universal metadata box format JUMBF encapsulation format, High Efficiency Image File Format (HEIF), MP4 file format, etc., and the embodiments of the present application do not limit this.

[0019] For example, the second data is encapsulated according to the JPEG JFIF encapsulation format. Among them, the second data can be encapsulated into the APPN field of the JPEG JFIF encapsulation format.

[0020] For another example, the second data is encapsulated into the payload of APP11 JP of the JUMBF encapsulation format.

[0021] For another example, the second data can be encapsulated in the box of the JUMBF encapsulation format.

[0022] For another example, the second data can be encapsulated in the box of the HEIF encapsulation format, and the box can be 'idat' or others.

[0023] For another example, the second data is encapsulated into the Supplemental Enhancement Information (SEI) of HEVC or VVC, or into the user-defined Network Abstract Layer (NAL) unit or the reserved NAL unit.

[0024] It should be understood that the second data can also be encapsulated into the file location indicated by the metadata location information (the metadata location information can be a kind of data of the metadata of the multimedia), such as the location after the end of image (EOI) of the complete JPEG file, or the end of the complete HEIF file, etc., and the embodiments of the present application do not limit this.

[0025] It should be understood that the embodiments of the present application do not limit the file position of the second data in the bitstream.

[0026] Exemplarily, the embodiments of the present application do not limit the metadata structure of the multimedia. For example, it may be the metadata structure corresponding to 'tmap' in 23008-12AMD 3.

[0027] Exemplarily, for example, the second data can be written into the bitstream in the form of an ASCII string; or, the second data can be written into the bitstream in the form of an ASCII code byte stream. It should be understood that the embodiments of the present application do not limit the form in which the second data is written into the bitstream.

[0028] According to the first aspect, the method further includes: generating second data based on the first data;

[0029] According to the first aspect, or any one of the above implementation manners of the first aspect, the method further includes: obtaining a floating-point data format; generating second data based on the first data, including: processing the first data according to the floating-point data format to obtain the second data.

[0030] Since the floating-point data format has a unified standard, after the encoding end processes the first data according to the floating-point data format to obtain the second data, the decoding end processes the second data according to the floating-point data format, and the first data can be restored; in this way, it can be ensured that the floating-point numbers finally obtained by the decoding end are the same as the original floating-point numbers at the encoding end.

[0031] Exemplarily, the first data can be converted into the second data according to the respective fields of the floating-point data specified by the floating-point data format, the layout and arithmetic interpretation of the respective fields, and the conversion method from decimal to binary.

[0032] In a possible manner, the floating-point data format can be preset; in this way, the preset floating-point data format can be obtained.

[0033] In a possible manner, the floating-point data format can be determined according to the application scenario, channel conditions, the significant digits of the first data, and so on.

[0034] According to the first aspect, or any one of the above implementation manners of the first aspect, the method further includes obtaining the bit width of the second data; processing the first data according to the floating-point data format to obtain the second data, including: processing the first data according to the floating-point data format and the bit width to obtain the second data.

[0035] Exemplarily, the bit width may refer to the number of bits occupied by data in a computer system. The bit width may include at least one of the following: 8bit, 16bit, 19bit, 32bit, or 64bit, etc., and the embodiments of the present application are not limited thereto.

[0036] In a possible way, the bit width can be preset; in this way, the preset bit width can be obtained.

[0037] In a possible way, the bit width can be determined according to the application scenario, channel conditions, and the significant digits of the first data, etc.

[0038] Specifically, the lengths of the respective fields of the floating-point data specified for the bit width in the floating-point data format can be determined; then, according to the respective fields of the floating-point data specified in the floating-point data format, the lengths of the respective fields specified for the bit width, the layout and arithmetic interpretation of the respective fields, and the way of converting decimal to binary, the first data can be converted into the second data.

[0039] According to the first aspect, or any one of the implementation manners of the above first aspect, the method further includes: writing the floating-point data format into the code stream.

[0040] Exemplarily, when the floating-point data format is determined according to the application scenario, channel conditions, and the significant digits of the first data, etc., the floating-point data format can be written into the code stream; so that the decoding end can obtain the floating-point data format and realize converting the second data into the first data.

[0041] Exemplarily, when the floating-point data format is preset and the floating-point data format has been synchronized in advance at the encoding and decoding ends, there is no need to write the floating-point data format into the code stream. Exemplarily, when the floating-point data format is preset, but the floating-point data format has not been synchronized in advance at the encoding and decoding ends, the floating-point data format can also be written into the code stream.

[0042] According to the first aspect, or any one of the implementation manners of the above first aspect, the method further includes: writing the bit width of the second data into the code stream.

[0043] Exemplarily, when the bit width is determined according to the application scenario, channel conditions, and the significant digits of the first data, etc., the bit width of the second data can be written into the code stream; so that the decoding end can obtain the bit width of the second data, and then can accurately divide a single second data from the multiple second data parsed from the code stream, and realize converting the second data into the first data.

[0044] Exemplarily, when the bit width is pre-set and the bit width has been synchronized at the encoding and decoding ends in advance, the bit width of the second data may not need to be written into the code stream. Exemplarily, when the bit width is pre-set but the bit width has not been synchronized at the encoding and decoding ends in advance, the bit width of the second data may also be written into the code stream.

[0045] It should be noted that the embodiments of the present application do not limit the bit width, the floating-point data format, and the position of the second data in the code stream. In one possible way, the bit width in the code stream is located after the floating-point data format, and the second data is located after the bit width.

[0046] According to the first aspect, or any implementation manner of the above first aspect, the second data is the binary representation or hexadecimal representation of the first data in the floating-point data format.

[0047] In one possible way, after processing the first data according to the floating-point data format and the bit width to obtain binary data, the binary data can be directly used as the second data. In one possible way, after processing the first data according to the floating-point data format and the bit width to obtain binary data, the binary data can be converted into hexadecimal data; the hexadecimal data is used as the second data.

[0048] For example, when the first data is "0.125490203" and the bit width of the second data is 32, the second data may be the hexadecimal representation of the first data in the IEEE 754 format, that is, "0X3E008081".

[0049] For example, when the first data is "0.35" and the bit width of the second data is 32, the second data may be the binary representation of the first data in the IEEE 754 format, that is, "01000000010100000000000000000000".

[0050] According to the first aspect, or any implementation manner of the above first aspect, the metadata of the multimedia is the metadata of the high dynamic range (HDR) image.

[0051] Exemplarily, the metadata of the HDR image may include dynamic metadata and static metadata. Among them, the dynamic metadata may refer to the data that changes over time, and the static metadata may refer to the data that does not change over time.

[0052] According to the first aspect, or any implementation manner of the above first aspect, the metadata of the multimedia is the metadata of the enhanced data, and the enhanced data is generated based on the standard dynamic range (SDR) image and the HDR image.

[0053] Among them, the metadata of the enhanced data is generated during the differential calculation of the base image and the first image to generate the enhanced data, and it affects the data of the differential calculation result (i.e., the enhanced data). In one possible case, the base image is an SDR image and the first image is an HDR image. In one possible case, the base image is an HDR image and the first image is an SDR image.

[0054] In this case, the bitstream may include the encoded base image and the encoded enhanced data. After receiving the bitstream, the decoding end can decode the bitstream to obtain the reconstructed image of the base image, the reconstruction result of the enhanced data, and the reconstructed metadata of the enhanced data. When the base image is an SDR image and the first image is an HDR image, if the decoding end supports displaying HDR, then according to the reconstructed metadata of the enhanced data, the reconstructed image of the base image and the reconstruction result of the enhanced data can be merged to obtain an HDR reconstructed image; if the decoding end only supports displaying SDR images, then only the reconstructed image of the base image needs to be displayed. When the base image is an HDR image and the first image is an SDR image, if the decoding end only supports displaying SDR, then according to the reconstructed metadata of the enhanced data, the reconstructed image of the base image and the reconstruction result of the enhanced data can be merged to obtain an SDR reconstructed image; if the decoding end supports displaying HDR images, then only the reconstructed image of the base image needs to be displayed. In this way, compatibility with different system support formats can be achieved.

[0055] Since the encoding and decoding method of the reconstructed metadata of the enhanced data in the embodiments of the present application will not cause loss of accuracy of the reconstructed metadata of the enhanced data, and the reconstructed metadata of the enhanced data obtained by the decoding end is the same as the metadata of the enhanced data encoded by the encoding end, the quality of the SDR image or HDR image obtained by the decoding end can be guaranteed.

[0056] According to the first aspect, or any one of the above implementation manners of the first aspect, the metadata of the multimedia may also be the metadata of audio data, or the metadata of image / video data in other formats, and the present application does not limit this.

[0057] According to the first aspect, or any one of the above implementation manners of the first aspect, the floating-point data format includes at least one of the following: IEEE 754, FP8 (E4M3), FP8 (E5M2), Bfloat16, TensorFloat-32.

[0058] For example, the second data is encapsulated according to the JPEG JFIF encapsulation format. Among them, the second data can be encapsulated into the APPN field of the JPEG JFIF encapsulation format.

[0059] According to the first aspect, or any implementation of the above first aspect, the bit width of the second data is greater than or equal to the bit width of the first data. In a computer, physical storage of floating-point numbers mostly adopts the form of exponent bits and mantissa bits. A floating-point number format represented by fewer exponent bits and mantissa bits (for example, 16 bits) can be losslessly converted into a floating-point number format represented by more exponent bits and mantissa bits (for example, 32 bits); in this application, by restricting the bit width of the second data to be greater than or equal to the bit width of the first data, lossless conversion from the first data to the second data can be ensured, thereby ensuring lossless transmission of metadata.

[0060] In a second aspect, an embodiment of the present application provides a bitstream, which is generated according to the first aspect or any implementation of the first aspect.

[0061] The second aspect and any implementation of the second aspect respectively correspond to the first aspect and any implementation of the first aspect. For the technical effects corresponding to the second aspect and any implementation of the second aspect, reference can be made to the technical effects corresponding to the first aspect and any implementation of the first aspect above, which will not be elaborated here.

[0062] In a third aspect, an embodiment of the present application provides a bitstream, the bitstream includes second data, the second data is a representation form of first data in a floating-point data format, and the first data is a floating-point data in the metadata of multimedia.

[0063] According to the third aspect, the bitstream further includes a floating-point data format.

[0064] According to the third aspect, the second data is obtained by processing the first data according to the floating-point data format and the bit width, and the bitstream further includes the bit width.

[0065] Exemplarily, when the metadata of multimedia is the metadata of enhanced data, the bitstream may further include an encoded base image and encoded enhanced data.

[0066] The third aspect and any implementation of the third aspect respectively correspond to the first aspect and any implementation of the first aspect. For the technical effects corresponding to the third aspect and any implementation of the third aspect, reference can be made to the technical effects corresponding to the first aspect and any implementation of the first aspect above, which will not be elaborated here.

[0067] In a fourth aspect, an embodiment of the present application provides a method for decoding metadata of multimedia, the method includes: First, receive a bitstream; Then, parse out second data from the bitstream, the second data is a representation form of first data in a floating-point data format, and the first data is a floating-point data in the metadata of multimedia; After that, generate the first data based on the second data.

[0068] Exemplarily, the second data can be parsed from the image encapsulation format in the bitstream.

[0069] For example, the second data can be parsed from the JPEG JFIF encapsulation format. For example, the second data can be parsed from the APPN field of the JPEG JFIF encapsulation format.

[0070] As another example, the second data can be parsed from the JUMBF encapsulation format. For example, the second data can be parsed from the payload of APP11 JP in the JUMBF encapsulation format. Another example is that the second data can be parsed from the box in the JUMBF encapsulation format.

[0071] As another example, the second data is parsed from the HEIF encapsulation format. For example, the second data can be parsed from the box in the HEIF encapsulation format. Another example is that the second data can be parsed from the 'idat' box in the HEIF encapsulation format.

[0072] As another example, the second data is parsed from the SEI of HEVC or VVC, or the user-defined NAL unit or the reserved NAL unit.

[0073] It should be understood that the metadata location information (the metadata location information can be a type of multimedia metadata, used to indicate the file location of the multimedia metadata) can also be parsed from the bitstream, and then the second data is parsed from the file location indicated by the metadata location information in the bitstream (for example, the location after the EOI (end of image) of the complete JPEG file, or the end of the complete HEIF file, etc., which is not limited in the embodiments of the present application).

[0074] It should be understood that the decoding end can also parse the second data from other file locations in the bitstream, which is not limited in the embodiments of the present application.

[0075] According to the fourth aspect, the method further includes: obtaining a floating-point data format; generating the first data based on the second data, including: processing the second data according to the floating-point data format to obtain the first data.

[0076] According to the fourth aspect, or any one of the implementation manners of the above fourth aspect, obtaining the floating-point data format includes: parsing the floating-point data format from the bitstream.

[0077] In a possible manner, when the decoding end fails to parse the floating-point data format from the bitstream, the pre-set floating-point data format can be obtained.

[0078] According to the fourth aspect, or any implementation manner of the above fourth aspect, the method further includes: obtaining the bit width of the second data; processing the second data in accordance with the floating-point data format to obtain the first data, including: processing the second data according to the floating-point data format and the bit width to obtain the first data.

[0079] According to the fourth aspect, or any implementation manner of the above fourth aspect, obtaining the bit width of the second data includes: parsing the bit width from the bit stream.

[0080] In a possible manner, when the decoding end fails to parse the bit width from the bit stream, a pre-set bit width may be obtained.

[0081] According to the fourth aspect, or any implementation manner of the above fourth aspect, the second data is the binary representation or hexadecimal representation of the first data in the floating-point data format.

[0082] According to the fourth aspect, or any implementation manner of the above fourth aspect, the metadata of the multimedia is the metadata of a high dynamic range (HDR) image.

[0083] According to the fourth aspect, or any implementation manner of the above fourth aspect, the metadata of the multimedia is the metadata of enhanced data, and the enhanced data is determined according to a standard dynamic range (SDR) image and a high dynamic range (HDR) image.

[0084] According to the fourth aspect, or any implementation manner of the above fourth aspect, the floating-point data format includes at least one of the following: IEEE 754, FP8 (E4M3), FP8 (E5M2), Bfloat16, TensorFloat-32.

[0085] The fourth aspect and any implementation manner of the fourth aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the fourth aspect and any implementation manner of the fourth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated here.

[0086] Fifth aspect, an embodiment of the present application provides an encoding device, which includes:

[0087] A metadata acquisition module, configured to acquire metadata of multimedia; wherein, the metadata of the multimedia includes first data, and the first data is floating-point data;

[0088] A data writing module, configured to write second data into a bit stream; wherein, the second data is the representation of the first data in the floating-point data format.

[0089] Exemplarily, the above encoding device can be used to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0090] The fifth aspect and any implementation manner of the fifth aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the fifth aspect and any implementation manner of the fifth aspect, reference can be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated here.

[0091] In a sixth aspect, an embodiment of the present application provides a decoding device, which includes:

[0092] a bitstream receiving module, configured to receive a bitstream;

[0093] a parsing module, configured to parse second data from the bitstream, where the second data is a representation form of first data in a floating-point data format, and the first data is floating-point data in the metadata of multimedia;

[0094] a second data generating module, configured to generate the first data based on the second data.

[0095] Exemplarily, the above decoding device can be used to execute the method in the fourth aspect or any possible implementation manner of the fourth aspect.

[0096] The sixth aspect and any implementation manner of the sixth aspect respectively correspond to the fourth aspect and any implementation manner of the fourth aspect. For the technical effects corresponding to the sixth aspect and any implementation manner of the sixth aspect, reference can be made to the technical effects corresponding to the fourth aspect and any implementation manner of the fourth aspect above, which will not be elaborated here.

[0097] In a seventh aspect, an embodiment of the present application provides an encoder, including: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the encoder is caused to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0098] The seventh aspect and any implementation manner of the seventh aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the seventh aspect and any implementation manner of the seventh aspect, reference can be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated here.

[0099] In an eighth aspect, an embodiment of the present application provides a decoder, including: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the decoder is caused to execute the method in the fourth aspect or any possible implementation manner of the fourth aspect.

[0100] The eighth aspect and any implementation manner of the eighth aspect respectively correspond to the fourth aspect and any implementation manner of the fourth aspect. For the technical effects corresponding to the eighth aspect and any implementation manner of the eighth aspect, reference may be made to the technical effects corresponding to the fourth aspect and any implementation manner of the fourth aspect above, which will not be elaborated here.

[0101] In a ninth aspect, an embodiment of the present application provides a decoder, including: a memory and a processor, the memory being coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the decoder executes the method in the first aspect or any possible implementation manner of the first aspect.

[0102] The ninth aspect and any implementation manner of the ninth aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the ninth aspect and any implementation manner of the ninth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated here.

[0103] In a tenth aspect, an embodiment of the present application provides a decoder, including: a memory and a processor, the memory being coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the decoder executes the method in the fourth aspect or any possible implementation manner of the fourth aspect.

[0104] The tenth aspect and any implementation manner of the tenth aspect respectively correspond to the fourth aspect and any implementation manner of the fourth aspect. For the technical effects corresponding to the tenth aspect and any implementation manner of the tenth aspect, reference may be made to the technical effects corresponding to the fourth aspect and any implementation manner of the fourth aspect above, which will not be elaborated here.

[0105] In an eleventh aspect, an embodiment of the present application provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the first aspect and any implementation manner of the first aspect are executed.

[0106] The eleventh aspect and any implementation manner of the eleventh aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the eleventh aspect and any implementation manner of the eleventh aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated here.

[0107] In a twelfth aspect, an embodiment of the present application provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, it enables the fourth aspect and any implementation manner of the fourth aspect.

[0108] The twelfth aspect and any implementation manner of the twelfth aspect respectively correspond to the fourth aspect and any implementation manner of the fourth aspect. For the technical effects corresponding to the twelfth aspect and any implementation manner of the twelfth aspect, reference may be made to the technical effects corresponding to the fourth aspect and any implementation manner of the fourth aspect above, which will not be elaborated here.

[0109] In a thirteenth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program runs on a computer or a processor, it enables the computer or the processor to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0110] The thirteenth aspect and any implementation manner of the thirteenth aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the thirteenth aspect and any implementation manner of the thirteenth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated here.

[0111] In a fourteenth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program runs on a computer or a processor, it enables the computer or the processor to execute the method in the fourth aspect or any possible implementation manner of the fourth aspect.

[0112] The fourteenth aspect and any implementation manner of the fourteenth aspect respectively correspond to the fourth aspect and any implementation manner of the fourth aspect. For the technical effects corresponding to the fourteenth aspect and any implementation manner of the fourteenth aspect, reference may be made to the technical effects corresponding to the fourth aspect and any implementation manner of the fourth aspect above, which will not be elaborated here.

[0113] In a fifteenth aspect, an embodiment of the present application provides a computer program product including computer instructions, and when the computer instructions are executed by a computer or a processor, it enables the computer or the processor to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0114] The fifteenth aspect and any implementation of the fifteenth aspect respectively correspond to the first aspect and any implementation of the first aspect. For the technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect, reference can be made to the technical effects corresponding to the first aspect and any implementation of the first aspect described above, which will not be elaborated here.

[0115] In a sixteenth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor is caused to execute the method in the fourth aspect or any possible implementation of the fourth aspect.

[0116] The sixteenth aspect and any implementation of the sixteenth aspect respectively correspond to the fourth aspect and any implementation of the fourth aspect. For the technical effects corresponding to the sixteenth aspect and any implementation of the sixteenth aspect, reference can be made to the technical effects corresponding to the fourth aspect and any implementation of the fourth aspect described above, which will not be elaborated here.

[0117] In a seventeenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores the bitstream in the third aspect or any implementation of the third aspect.

[0118] The seventeenth aspect and any implementation of the seventeenth aspect respectively correspond to the third aspect and any implementation of the third aspect. For the technical effects corresponding to the seventeenth aspect and any implementation of the seventeenth aspect, reference can be made to the technical effects corresponding to the third aspect and any implementation of the third aspect described above, which will not be elaborated here.

[0119] In an eighteenth aspect, an embodiment of the present application provides a device for storing a bitstream. The device includes: a receiver and at least one storage medium. The receiver is configured to receive the bitstream; the at least one storage medium is configured to store the bitstream in the third aspect or any implementation of the third aspect.

[0120] The eighteenth aspect and any implementation of the eighteenth aspect respectively correspond to the third aspect and any implementation of the third aspect. For the technical effects corresponding to the eighteenth aspect and any implementation of the eighteenth aspect, reference can be made to the technical effects corresponding to the third aspect and any implementation of the third aspect described above, which will not be elaborated here.

[0121] In a nineteenth aspect, an embodiment of the present application provides an apparatus for transmitting a bitstream. The apparatus includes: a transmitter and at least one storage medium. The at least one storage medium is configured to store the bitstream in the third aspect or any implementation manner of the third aspect; the transmitter is configured to obtain the bitstream from the storage medium and send the bitstream to an end-side device through a transmission medium.

[0122] The nineteenth aspect and any implementation manner of the nineteenth aspect respectively correspond to the third aspect and any implementation manner of the third aspect. For the technical effects corresponding to the nineteenth aspect and any implementation manner of the nineteenth aspect, reference may be made to the technical effects corresponding to the third aspect and any implementation manner of the third aspect, which will not be elaborated herein.

[0123] In a twentieth aspect, an embodiment of the present application provides a system for distributing a bitstream. The system includes: at least one storage medium configured to store the bitstream in the third aspect or any implementation manner of the third aspect, and a streaming media device configured to obtain a target bitstream from the at least one storage medium and send the target bitstream to an end-side device, where the streaming media device includes a content server or a content distribution server.

[0124] The twentieth aspect and any implementation manner of the twentieth aspect respectively correspond to the third aspect and any implementation manner of the third aspect. For the technical effects corresponding to the twentieth aspect and any implementation manner of the twentieth aspect, reference may be made to the technical effects corresponding to the third aspect and any implementation manner of the third aspect, which will not be elaborated herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0125] Figure 1A It is a schematic diagram of an application scenario shown exemplarily;

[0126] Figure 1B It is a schematic diagram of an end-to-end process of an HDR video shown exemplarily;

[0127] Figure 1C It is a schematic diagram of an encoding process of an image and metadata of the image shown exemplarily;

[0128] Figure 1D It is a schematic diagram of a decoding process of an image and metadata of the image shown exemplarily;

[0129] Figure 2 It is a schematic diagram of an encoding process 200 of metadata shown exemplarily;

[0130] Figure 3 It is a schematic diagram of a decoding process 300 of metadata shown exemplarily;

[0131] Figure 4A It is a schematic diagram of an encoding process 400 of metadata shown exemplarily;

[0132] Figure 4B Schematic diagram of a floating-point data format shown by way of example;

[0133] Figure 4C Schematic diagram of a floating-point data format shown by way of example;

[0134] Figure 5A Schematic diagram of a bitstream structure shown by way of example;

[0135] Figure 5B Schematic diagram of a bitstream structure shown by way of example;

[0136] Figure 5C Schematic diagram of a bitstream structure shown by way of example;

[0137] Figure 5D Schematic diagram of a bitstream structure shown by way of example;

[0138] Figure 6 Schematic diagram of the decoding process 600 of metadata shown by way of example;

[0139] Figure 7 Schematic diagram of the encoding process 700 of an image and the metadata of the image shown by way of example;

[0140] Figure 8 Schematic diagram of the decoding process 800 of an image and the metadata of the image shown by way of example;

[0141] Figure 9 Schematic diagram of the decoding process 900 of an image and the metadata of the image shown by way of example;

[0142] Figure 10 Schematic diagram of the encoding device 1000 shown by way of example;

[0143] Figure 11 Schematic diagram of the decoding device 1100 shown by way of example;

[0144] Figure 12 Schematic block diagram of a codec system applied in the embodiments of the present application;

[0145] Figure 13A Block diagram of a content providing system for implementing a content distribution service applied in the embodiments of the present application;

[0146] Figure 13B For Figure 13A Schematic diagram of the exemplary structure of the terminal device 2106 in;

[0147] Figure 14A Schematic diagram of the working process of a streaming media system applied in the embodiments of the present application;

[0148] Figure 14B Schematic diagram of a streaming media system architecture applied to the embodiments of the present application;

[0149] Figure 15A Schematic diagram of a possible system architecture applicable to the embodiments of the present application;

[0150] Figure 15B Schematic diagram of the structure of an image processing system provided by the embodiments of the present application;

[0151] Figure 16 Schematic diagram of the structure of the device shown by way of example. Detailed implementation manners

[0152] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0153] In this document, the term "and / or" is merely used to describe the associated relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. These three situations.

[0154] The terms "first", "second", etc. in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, rather than to describe the specific order of the target objects.

[0155] In the embodiments of the present application, words such as "exemplarily" or "for example" are used to give examples, illustrations or explanations. Any embodiment or design solution described as "exemplarily" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplarily" or "for example" aims to present relevant concepts in a specific manner.

[0156] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" refers to two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0157] Exemplarily, the multimedia involved in the present application may include, but is not limited to: images, videos, and audio.

[0158] Exemplarily, the present application can be applied to various image services (such as taking photos with a mobile phone, cloud photo albums, etc.) scenarios, various video / audio service scenarios (such as live streaming, video on demand, etc.), and so on. The embodiments of the present application are not limited thereto. The present application will be described by taking multimedia as an example of an image.

[0159] Figure 1A It is a schematic diagram of an exemplary application scenario shown.

[0160] Referring to Figure 1A , exemplarily, the first electronic device may include an image generation module, a metadata generation module, an encoding module (or encoder), and a sending module.

[0161] Exemplarily, the encoding module can be a software module or a hardware module. The embodiments of the present application are not limited thereto.

[0162] Exemplarily, the image generation module may include, but is not limited to, an image acquisition module, an image editing module, etc. The embodiments of the present application are not limited thereto.

[0163] It should be understood that Figure 1A is only an example of the first electronic device. The first electronic device in some other embodiments of the present application has more or fewer modules than Figure 1A shown. The embodiments of the present application are not limited thereto.

[0164] Referring to Figure 1A , exemplarily, the second electronic device may include a display module, a decoding module (or decoder), and a receiving module. Exemplarily, the decoding module can be a software module or a hardware module. The embodiments of the present application are not limited thereto. It should be understood that Figure 1A is only an example of the second electronic device. The second electronic device in some other embodiments of the present application has more modules than Figure 1A shown. The embodiments of the present application are not limited thereto.

[0165] Continuing to refer to Figure 1A, Exemplarily, after the image generation module of the first electronic device generates an image, the generated image can be output to the encoding module and the metadata generation module. Subsequently, the metadata generation module can generate the metadata of the image and output the metadata of the image to the encoding module. After that, the encoding module can encode the image and the metadata of the image to obtain a bitstream (wherein, the bitstream can also be referred to as a bit stream or a file stream or a file), and output the encoded bitstream to the sending module. Then, the sending module can send the bitstream to the second electronic device. After the receiving module of the second electronic device receives the bitstream, the bitstream can be output to the decoding module; then, the decoding module can decode the bitstream to obtain the reconstructed image and the reconstructed metadata of the image, and output the reconstructed image and the reconstructed metadata of the image to the display module, and the display module can display the reconstructed image according to the reconstructed metadata of the image.

[0166] Exemplarily, the first electronic device includes but is not limited to: a server, a PC (Personal Computer), a laptop, a tablet, a mobile phone, a watch.

[0167] Exemplarily, the second electronic device includes but is not limited to: a PC, a laptop, a tablet, a mobile phone, a watch.

[0168] Exemplarily, when the first electronic device is used for encoding and the second electronic device is used for decoding, the first electronic device can be referred to as the encoding end, and the second electronic device can be referred to as the decoding end. When the second electronic device is used for encoding and the first electronic device is used for decoding, the second electronic device can be referred to as the encoding end, and the first electronic device can be referred to as the decoding end.

[0169] In a possible way, Figure 1A the image encoded by the first electronic device can be an HDR image. For the convenience of understanding, SDR and HDR are introduced first.

[0170] Dynamic Range is used in many fields to represent the ratio of the maximum value to the minimum value of a certain variable. In digital images, the dynamic range represents the ratio between the maximum gray value and the minimum gray value within the range that the image can be displayed. The dynamic range of nature is quite large. The brightness of the night scene under the starry sky is about 0.001 cd / m2, and the brightness of the sun itself is as high as 1,000,000,000 cd / m2. Such a dynamic range reaches 1,000,000,000 / 0.001 = 10 13 orders of magnitude. However, in the real scenes of nature, the brightness of the sun and the brightness of the starlight cannot be obtained simultaneously. For the natural scenes in the real world, the dynamic range is between 10 -3 and 10 6Within this range. In most current color digital images, each of the R, G, and B channels uses one byte (8 bits) to store. That is to say, the representation range of each channel is 0 to 255 gray levels. Here, 0 to 255 is the dynamic range of the image. Since the dynamic range in the same scene in the real world is between 10 -3 and 10 6 within this range, we call it high dynamic range (HDR). In contrast, the dynamic range of ordinary pictures is low dynamic range (LDR). The imaging process of a digital camera is actually a mapping from the high dynamic range of the real world to the low dynamic range of the photo.

[0171] Standard dynamic range images correspond to high dynamic range images. The 8-bit images in traditional formats such as jpeg that we commonly use can be considered standard dynamic range images. Before the emergence of cameras capable of shooting HDR images, traditional cameras could only record the light information captured within a certain range by controlling the exposure value. Since the maximum illuminance information of the display device cannot reach the luminance information of the real world, and we view images through the display device, an electro-optical transfer function is needed. Early display devices used cathode ray tube (CRT) monitors, and its electro-optical transfer function was the Gamma function. This electro-optical transfer function based on the "Gamma" function was defined in the ITU-R Recommendation BT.1886 standard.

[0172] However, with the upgrade of display devices, the illuminance range of display devices has been continuously increasing. The current consumer-grade HDR monitors have an illuminance of 600 cd / m2, and the illuminance information of high-end HDR monitors can reach 2000 cd / m2, far exceeding the illuminance information of SDR display devices. The electro-optical conversion function in the ITU-R Recommendation BT.1886 standard cannot well express the display performance of HDR display devices. Therefore, an improved electro-optical transfer function is needed to adapt to the upgrade of display devices. The idea of the electro-optical transfer function comes from the mapping function in the Tone Mapping algorithm. Appropriate adjustment of the mapping function is the electro-optical transfer function. Currently, the following three common electro-optical conversion functions are available: Perception Quantization (PQ) function, gamma, logarithm (log) function, Hybrid Log-Gamma (HLG) function, etc.; these three electro-optical conversion functions are the conversion functions specified by the AVS standard.

[0173] Among them, if a better experience is required for HDR images and videos, it is an end-to-end process, which can be referred toFigure 1B , Figure 1B describes the end-to-end process of HDR video.

[0174] Since the dynamic range of the image acquisition module in a specific shooting state is fixed, in order to obtain an image with a higher dynamic range, generally multiple images with different exposures collected at the same moment are used to synthesize a multi-exposure image, and then a higher-dynamic-range image, that is, an HDR image, is obtained. The bit width of the HDR image obtained in this way is usually greater than 10 bits, so it can accommodate scenes with a higher dynamic range.

[0175] Referring to Figure 1B , exemplarily, material production may refer to the process of using multiple images with different exposures collected at the same moment to synthesize an HDR image; and synthesizing temporally consecutive HDR images to form an original HDR video; the original HDR video obtained through material production can be referred to as material.

[0176] After that, the HDR image / video production module of the image generation module can perform processes such as editing / color correction on the material to obtain the final HDR video (that is, the Master). Subsequently, the dynamic metadata generation module can generate dynamic metadata based on the final HDR video. Then, the encoding module can encode the HDR video and the dynamic metadata to obtain a bitstream and distribute / transmit the bitstream through the network. After the decoding end receives the bitstream, it can call the decoding module to decode the bitstream to obtain the reconstructed HDR video and the reconstructed dynamic metadata. After that, the display module can display the reconstructed HDR video according to the reconstructed dynamic metadata.

[0177] However, some codec standards can support at most the codec of images with a bit width of 10 bits, so the codec of HDR images cannot be implemented; in addition, the standard dynamic range (SDR) transcoding device cannot encapsulate the metadata of HDR images into the bitstream of SDR images, resulting in the inability of terminal devices receiving the SDR bitstream to restore HDR images. In view of this, the embodiments of the present application provide a method for encoding and decoding an image and the metadata of the image, which can be referred to Figure 1C and Figure 1D .

[0178] Figure 1C is a schematic diagram of the encoding process of an exemplary image and the metadata of the image.

[0179] Referring to Figure 1C, Exemplarily, the encoding end may use the SDR image as the base image and the HDR image as the first image; or, use the HDR image as the base image and the SDR image as the first image. The embodiments of the present application do not limit this. Next, the difference between the base image and the first image can be calculated to obtain enhancement data and metadata. After that, the base image, the enhancement data, and the metadata can be encoded to obtain a bitstream.

[0180] Figure 1D Schematic diagram of the decoding process of the exemplary image and the metadata of the image.

[0181] Refer to Figure 1D , Exemplarily, after receiving the bitstream, the decoding end can decode the bitstream to obtain the reconstructed image of the base image, the reconstruction result of the enhancement data, and the reconstructed metadata. When the base image is an SDR image and the first image is an HDR image, if the decoding end supports HDR display, the reconstructed metadata can be used to merge the reconstructed image of the base image and the reconstruction result of the enhancement data to obtain an HDR reconstructed image; if the decoding end only supports SDR image display, only the reconstructed image of the base image needs to be displayed. When the base image is an HDR image and the first image is an SDR image, if the decoding end only supports SDR display, the reconstructed metadata can be used to merge the reconstructed image of the base image and the reconstruction result of the enhancement data to obtain an SDR reconstructed image; if the decoding end supports HDR image display, only the reconstructed image of the base image needs to be displayed.

[0182] In this way, compatibility with different system support formats can be achieved.

[0183] The following describes the encoding and decoding processes of the metadata.

[0184] Figure 2 Schematic diagram 200 of the encoding process of the exemplary metadata.

[0185] S201, Obtain the metadata of the multimedia. The metadata of the multimedia includes first data, and the first data is floating-point data.

[0186] In one possible way, the metadata of the multimedia can be the metadata of the enhancement data. Among them, the metadata of the enhancement data is generated during the process of calculating the difference between the base image and the first image to generate the enhancement data, and the data that affects the difference calculation result (i.e., the enhancement data); it will be specifically described in the subsequent embodiments.

[0187] In one possible way, the metadata of the multimedia can be the metadata of the HDR image, which can include dynamic metadata and static metadata. Among them, the dynamic metadata can refer to the data that changes over time, and the static metadata can refer to the data that does not change over time.

[0188] It should be understood that the metadata of the multimedia may also be the metadata of other images, the metadata of audio, the metadata of video, etc., and the embodiments of the present application do not limit this.

[0189] Exemplarily, the metadata of the multimedia may include first data, and the first data may be floating-point type (including 32-bit float type or 64-bit double type) data; for example, the first data is "0.125490203".

[0190] It should be understood that the metadata of the multimedia may also include other data in addition to the first data, and the other data in the metadata of the multimedia can be encoded and decoded by using the methods of the prior art, and the embodiments of the present application do not limit this; the present application describes the encoding and decoding process of the floating-point type data in the multimedia data.

[0191] Exemplarily, there may be one or more first data.

[0192] S202, write the second data into the bitstream; wherein, the second data is the representation form of the first data in the floating-point data format.

[0193] Exemplarily, the first data can be processed to obtain the representation form of the first data in the floating-point data format (which can be called the second data hereinafter).

[0194] Exemplarily, the second data may be the binary representation form or the hexadecimal representation form of the first data in the floating-point data format; the embodiments of the present application do not limit this.

[0195] Exemplarily, the floating-point data format is a data format that stipulates each field constituting the floating-point type data, the layout of each field, and the arithmetic interpretation. The floating-point data format may include but is not limited to: IEEE 754, FP8 (E4M3), FP8 (E5M2), Bfloat16, TensorFloat-32; it should be understood that other floating-point data formats may also be included, and the embodiments of the present application do not limit this.

[0196] For example, when the first data is "0.125490203" and the bit width is 32, the second data may be the hexadecimal representation of the first data in the IEEE754 format, that is, "0X3E008081".

[0197] For example, when the first data is "0.35" and the bit width is 32, the second data may be the binary representation of the first data in the IEEE 754 format, that is, "01000000010100000000000000000000".

[0198] Exemplarily, there may be one or more second data. Based on one first data, one second data can be generated; that is, multiple first data and multiple second data are in one-to-one correspondence.

[0199] Exemplarily, after the second data is obtained, the second data can be written into the code stream. For example, the second data can be written into the code stream in the form of an ASCII string; or, the second data can be written into the code stream in the form of an ASCII code byte stream. It should be understood that the embodiments of the present application do not limit the form in which the second data is written into the code stream.

[0200] Figure 3 Schematic diagram of the decoding process 300 of the metadata shown by way of example. Figure 3 in the decoding process 300 of the metadata is corresponding to Figure 2 the encoding process 200 of the metadata in

[0201] S301, Receive the code stream.

[0202] Exemplarily, after the encoding end sends the code stream to the decoding end, the decoding end can receive the code stream.

[0203] S302, Parse out the second data from the code stream. The second data is a representation form of the first data in floating-point data format, and the first data is a floating-point data in the metadata of the multimedia.

[0204] Exemplarily, the code stream received by the decoding end may include the second data, and thus, the second data can be parsed out from the code stream.

[0205] S303, Generate the first data based on the second data.

[0206] Exemplarily, after the second data is obtained, the second data can be processed to convert the second data into floating-point data, that is, the first data can be obtained. Exemplarily, the first data obtained in S303 can be Figure 1C part or all of the reconstructed metadata in

[0207] For example, if the second data is "0X3E008081", the first data generated based on the second data can be "0.125490203".

[0208] For another example, if the second data is "01000000010100000000000000000000", the first data generated based on the second data can be "0.35".

[0209] For a specific syntax element, a codeword of a conventional length is usually transmitted in the bitstream. Since floating-point numbers are usually numbers with high precision, during the encapsulation process of syntax elements in existing standards, they will be fixed-pointized or converted into other formats (such as ASCII strings, and length truncation is involved in the conversion process), resulting in loss of precision. However, in this application, the form of storing floating-point numbers in computer memory (i.e., the floating-point data format of the second data) is used for transmission in the bitstream, which can ensure that the floating-point number values stored in the memory of the sending end and the receiving end are exactly the same. In this way, the floating-point data obtained by the decoding end through the conversion of the second data is the same as the first data used for encoding at the encoding end, thus achieving the consistency between the floating-point number finally obtained at the decoding end and the original floating-point number at the encoding end.

[0210] Secondly, since the floating-point data in the metadata of the multimedia at the encoding and decoding ends is consistent, in scenarios where it is necessary to generate reconstructed multimedia based on the metadata of the multimedia or display the reconstructed multimedia content according to the metadata of the multimedia, the quality of the reconstructed multimedia content or the visual experience when displaying the reconstructed multimedia content can be improved.

[0211] The encoding and decoding processes of the metadata are specifically described below.

[0212] FIG. 4 is a schematic diagram of an exemplary encoding process 400 of metadata.

[0213] S401, obtain the metadata of the multimedia, where the metadata of the multimedia includes first data, and the first data is floating-point data.

[0214] Exemplarily, for each first data in the metadata of the multimedia, the encoding can be performed according to the steps of S402 to S407.

[0215] S402, obtain the floating-point data format.

[0216] In a possible way, the floating-point data format can be preset; in this way, the preset floating-point data format can be obtained.

[0217] In a possible way, the floating-point data format can be determined according to the application scenario, channel conditions, significant bits of the first data, etc.

[0218] S403, obtain the bit width of the second data.

[0219] Exemplarily, the bit width can refer to the number of bits (bit) occupied by the data in the computer system. The bit width can include at least one of the following: 8bit, 16bit, 19bit, 32bit or 64bit, etc., and the embodiments of this application do not limit this.

[0220] In a possible way, the bit width can be preset; in this way, the preset bit width can be obtained.

[0221] In a possible way, the bit width can be determined according to the application scenario, channel conditions, the effective number of bits of the first data, etc.

[0222] Exemplarily, the combinations of floating-point data formats and bit widths can be as shown in Table 1 below:

[0223] Table 1

[0224] Floating-point data format Bit width (bit) IEEE754 16 IEEE754 32 IEEE754 64 FP8 (E4M3) 8 FP8 (E5M2) 8 Half-precision 16 Bfloat16 16 TensorFloat-32 19 Single-precision 32

[0225] It should be noted that the embodiments of the present application do not limit the execution order of S402 and S403.

[0226] S404, process the first data according to the floating-point data format and the bit width to obtain the second data.

[0227] Exemplarily, the first data can be converted according to the floating-point data format and the bit width to obtain the second data. Specifically, the lengths of the respective fields of the floating-point data specified by the floating-point data format for the bit width can be determined; thereafter, according to the respective fields of the floating-point data, the lengths of the respective fields, the layout and arithmetic interpretation of the respective fields, and the method of converting decimal to binary, the first data can be converted into the second data.

[0228] For example, if the floating-point data format is IEEE 754 and the bit width is 16 bits, correspondingly, the floating-point data format can be referred to Figure 4B (1) shown. Among them, from left to right, the 1st bit is used to represent the sign of the floating-point data, the 2nd to 6th bits are used to represent the exponent of the floating-point data, and the 7th to 16th bits are used to represent the fractional part of the floating-point data. According to the method of converting decimal to binary, converting the first data "3.25" into the second data "0100000001010000" can be as Figure 4B (2) shown.

[0229] For example, if the floating-point data format is IEEE 754 and the bit width is 32 bits, correspondingly, the floating-point data format can be referred to Figure 4C(1) As shown. Among them, from left to right, the 1st bit is used to represent the sign of the floating-point data, the 2nd to 9th bits are used to represent the exponent of the floating-point data, and the 9th to 32nd bits are used to represent the fractional part of the floating-point data. According to the method of converting decimal to binary, the first data "3.25" is converted into the second data "01000000010100000000000000000000", which can be as Figure 4C (2) shown.

[0230] It should be noted that in one possible way, after processing the first data according to the floating-point data format and bit width to obtain the binary data, the binary data can be directly used as the second data. In one possible way, after processing the first data according to the floating-point data format and bit width to obtain the binary data, the binary data can be converted into hexadecimal data; and the hexadecimal data is used as the second data.

[0231] Exemplarily, other floating-point data formats can be specifically referred to the descriptions in the corresponding standards, which will not be elaborated here.

[0232] S405, write the floating-point data format into the bitstream.

[0233] Exemplarily, when the floating-point data format is determined according to the application scenario, channel conditions, the effective number of bits of the first data, etc., the floating-point data format can be written into the bitstream; so that the decoding end can obtain the floating-point data format and realize converting the second data into the first data.

[0234] Exemplarily, when the floating-point data format is pre-set and the floating-point data format has been synchronized in advance between the encoding and decoding ends, there is no need to write the floating-point data format into the bitstream. Exemplarily, when the floating-point data format is pre-set, but the floating-point data format has not been synchronized in advance between the encoding and decoding ends, the floating-point data format can also be written into the bitstream.

[0235] S406, write the bit width of the second data into the bitstream.

[0236] Exemplarily, when the bit width is determined according to the application scenario, channel conditions, the effective number of bits of the first data, etc., the bit width of the second data can be written into the bitstream; so that the decoding end can obtain the bit width of the second data, and then can accurately divide a single second data from multiple second data parsed from the bitstream, and realize converting the second data into the first data.

[0237] Exemplarily, when the bit width is pre-set and the bit width has been synchronized at the encoding and decoding ends in advance, it is not necessary to write the bit width of the second data into the code stream. Exemplarily, when the bit width is pre-set but the bit width has not been synchronized at the encoding and decoding ends in advance, the bit width of the second data can also be written into the code stream.

[0238] It should be noted that the embodiments of the present application do not limit the positions of the bit width and the floating-point data format in the code stream.

[0239] S407, write the second data into the code stream.

[0240] Exemplarily, when the multimedia is an image or a video, the second data can be encapsulated according to the image encapsulation format to obtain a code stream. Among them, the image encapsulation format includes but is not limited to: JPEG JFIF encapsulation format, JPEG general metadata box format JUMBF encapsulation format, High Efficiency Image File Format (HEIF), MP4 file format, etc., and the embodiments of the present application do not limit this.

[0241] For example, encapsulate the second data according to the JPEG JFIF encapsulation format. Among them, the second data can be encapsulated into the APPn field of the JPEG JFIF encapsulation format, as shown in Table 2:

[0242] Table 2

[0243]

[0244] Among them, the explanation of the information shown in Table 1 can be referred to the description in ISO / IEC FDIS 10918-4:2023 APPn markers (JPEG 1 part 4), and will not be elaborated here.

[0245] For example, encapsulate the second data according to the JUMBF encapsulation format.

[0246] In a possible way, the second data can be encapsulated into the payload of APP11 JP of the JUMBF encapsulation format, as shown in Table 3 below:

[0247] Table 3

[0248] Parameter Size (bits) Value APP11 16 0xFFEB Le 16 8..65535 Cl 16 0x4A50 (ASCII encoding of "JP”) En 16 1..65535 Z 32 1..232-1 Lbox 32 1or8..232-1 Tbox 32 0..232-1 XLBox 0or64 16..264-1 Payload Data Varies Varies

[0249] Among them, the explanation of the information shown in Table 2 can be referred to the description in the JPEG general metadata box format standard, and will not be elaborated here.

[0250] In a possible way, the second data can be encapsulated in a box of the JUMBF encapsulation format.

[0251] For example, the second data can be encapsulated in HEIF. For example, the second data can be encapsulated into the packet structure of HEIF. Still for example, the second data is encapsulated into the 'tmap' and 'idat' of HEIF; among them, the descriptions of 'tmap' and 'idat' can refer to the descriptions in ISO / IEC 23008-12 AMD 3, which will not be elaborated here.

[0252] Exemplarily, when the multimedia is an image or a video, the second data can be encapsulated into the Supplemental Enhancement Information (SEI) of HEVC or VVC, or into a user-defined Network Abstract Layer (NAL) unit or a reserved NAL unit.

[0253] It should be understood that the second data can also be encapsulated into the file location indicated by the metadata location information (the metadata location information can be a type of data in the metadata of the multimedia), such as the location after the end of image (EOI) of a complete JPEG file, or the end of a complete HEIF file, etc. The embodiments of the present application do not limit this.

[0254] It should be understood that the embodiments of the present application do not limit the file location of the second data in the bitstream.

[0255] Exemplarily, the embodiments of the present application do not limit the metadata structure of the multimedia. For example, it can be the metadata structure corresponding to 'tmap' in 23008-12 AMD 3, as shown below:

[0256]

[0257] Among them, SignedRational represents the metadata type, and ain_map_min, …… alternate_offset are the metadata. Some or all of these metadata are the first data.

[0258] Still for example, the metadata structure of the multimedia can be described in XML or in XML format, as shown below:

[0259]

[0260] Among them, PerComponentMinGainmapValues, …… AlternateHDRheadroom are the metadata. Some or all of these metadata are the first data.

[0261] In addition, when writing the second data into the bitstream in the form of an ASCII - code byte stream, the second data can be written into the bitstream in the order of high - order bits first and low - order bits second, or in the order of low - order bits first and high - order bits second. The embodiments of the present application do not limit this.

[0262] It should be noted that the embodiments of the present application do not limit the execution order of S405, S406, and S407. For example, it can be executed in the order of S405→S406→S407; correspondingly, in the bitstream, the bit width is located after the floating - point data format, and the second data is located after the bit width.

[0263] Figure 5A Schematic diagram of the bitstream structure shown for illustration. As Figure 5A shown, in one possible way, the bitstream of the present application may include the second data.

[0264] Figure 5B Schematic diagram of the bitstream structure shown for illustration. As Figure 5B shown, in one possible way, the bitstream of the present application may include the second data and the floating - point data format.

[0265] Figure 5C Schematic diagram of the bitstream structure shown for illustration. As Figure 5C shown, in one possible way, the bitstream of the present application may include the second data and the bit width.

[0266] Figure 5D Schematic diagram of the bitstream structure shown for illustration. As Figure 5D shown, in one possible way, the bitstream of the present application may include the second data, the floating - point data format, and the bit width.

[0267] It should be noted that the embodiments of the present application do not limit the order / arrangement position of the floating - point data format, the bit width, and the second data in the bitstream.

[0268] Exemplarily, the embodiments of the present application provide a device for transmitting a bitstream. The device for transmitting the bitstream may include: a transmitter and at least one storage medium. The at least one storage medium is used to store the bitstream generated by the encoding process 200 of the above - mentioned metadata or the encoding process 400 of the metadata; the transmitter is used to obtain the bitstream from the storage medium and send the bitstream to the terminal - side device through the transmission medium.

[0269] Exemplarily, a device for encoding an end or a transport stream can send the stream to a system for distributing the stream. The system for distributing the stream may include: at least one storage medium and a streaming media device; the storage medium is used to store the stream generated by the encoding process 200 of the metadata or the encoding process 400 of the metadata; the streaming media device is used to obtain a target stream from at least one storage medium and send the target stream to an end-side device, where the streaming media device includes a content server or a content distribution server.

[0270] Exemplarily, an embodiment of the present application further provides a device for storing a stream. The device for storing the stream may include: a receiver and at least one storage medium, the receiver is used to receive the stream; at least one storage medium is used to store the stream. The stream may be the stream generated by the encoding process 200 of the metadata or the encoding process 400 of the metadata.

[0271] Figure 6 Schematic diagram of the decoding process 600 of the metadata shown exemplarily. Figure 6 In the decoding process 600 of the metadata Figure 4A corresponds to the encoding process 400 of the metadata.

[0272] S601, receive the stream.

[0273] S602, parse out the second data, floating-point data format, and bit width of the second data from the stream.

[0274] Exemplarily, when the encoding end writes the floating-point data format and the bit width of the second data into the stream, the decoding end can parse out the second data, floating-point data format, and bit width from the stream.

[0275] Exemplarily, the second data can be parsed out from the image encapsulation format in the stream.

[0276] For example, the second data can be parsed out from the JPEG JFIF encapsulation format. Such as parsing out the second data from the APPN field of the JPEG JFIF encapsulation format.

[0277] For another example, the second data can be parsed out from the JUMBF encapsulation format. Such as, the second data can be parsed out from the payload of APP11 JP of the JUMBF encapsulation format. For another example, the second data can be parsed out from the box of the JUMBF encapsulation format.

[0278] For another example, the second data is parsed out from the SEI of HEVC or VVC, or a user-defined NAL unit or a reserved NAL unit.

[0279] It should be understood that the metadata location information (which can be a type of metadata of the multimedia and is used to indicate the file location of the metadata of the multimedia) can also be parsed from the bitstream, and then the second data can be parsed from the file location indicated by the metadata location information in the bitstream (for example, the location after the EOI (end of image) of the complete JPEG file, or the end of the complete HEIF file, etc., and the embodiments of the present application do not limit this).

[0280] It should be understood that the decoding end can also parse the second data from other file locations in the bitstream, and the embodiments of the present application do not limit this.

[0281] Exemplarily, when the floating-point data format cannot be parsed from the bitstream by the decoding end, the pre-set floating-point data format can be obtained.

[0282] Exemplarily, when the bit width cannot be parsed from the bitstream by the decoding end, the pre-set bit width can be obtained.

[0283] Exemplarily, multiple second data can be parsed from the bitstream, and these multiple second data are spliced together; furthermore, the multiple spliced second data can be divided according to the floating-point data format, bit width, and metadata structure to obtain individual independent second data.

[0284] S603, process the second data according to the floating-point data format and bit width to obtain the first data.

[0285] Exemplarily, each second data can be converted into floating-point data according to the floating-point data format and bit width to obtain a first data. Specifically, the process of converting the representation in the floating-point data format into floating-point data can refer to the description of the prior art and will not be elaborated here.

[0286] Based on the above encoding and decoding process of the metadata, the generation process and encoding and decoding process of the base image, enhancement data, and metadata of the enhancement data will be described below.

[0287] Figure 7 It is a schematic diagram of the encoding process 700 of the exemplary image and the metadata of the image. The metadata of the image in the encoding process 700 is the metadata of the enhancement data.

[0288] S701, obtain the base image.

[0289] In a possible way, an SDR image can be obtained and used as the base image.

[0290] In a possible way, an HDR image can be obtained and used as the base image.

[0291] It should be noted that the foreground of the SDR image is the same as that of the HDR image, and the background of the SDR image is the same as that of the HDR image. Or rather, the SDR image and the HDR image contain the same content (including objects, scenes, etc.).

[0292] Exemplarily, the way to obtain the HDR image can be as follows:

[0293] For example, multiple SDR images with different exposure values can be collected by a camera, and then, these multiple SDR images with different exposure values are used to synthesize the HDR image. Among them, the photosensitive element of the camera has different optoelectronic characteristics at different positions, and thus multiple SDR images with different exposure values can be collected.

[0294] Also for example, the HDR image can be captured by a professional acquisition device; then, the HDR image is transmitted to the encoding end by means of sending or copying, etc.

[0295] It should be noted that the embodiments of the present application do not limit the way to obtain the HDR image.

[0296] It should be noted that the embodiments of the present application do not limit the format of the HDR image. For example, in terms of color space, the HDR image can be a YUV image or an RGB image. In terms of the bit width of the data, the bit width of the HDR image can be 8bit, 10bit, or 12bit, etc. The optoelectronic conversion function corresponding to the HDR image can be a Perception Quantization (PQ) function, gamma, logarithm (log) function, Hybrid Log-Gamma (HLG) function, etc.

[0297] Exemplarily, the process of obtaining the SDR image can be as follows:

[0298] For example, an SDR image with different light and dark characteristics different from the HDR image is collected by a camera, or an SDR image in a format adapted to the display characteristics of existing devices is collected by a camera.

[0299] Also for example, the SDR image is captured by a professional acquisition device, and then, the SDR image is transmitted to the encoding end by means of sending or copying, etc.

[0300] Also for example, the SDR image is generated according to the HDR image. For example, the SDR image can be obtained by tone mapping the HDR image. Also for example, the HDR image is input into a neural network to obtain the SDR image.

[0301] It should be noted that the embodiments of the present application do not limit the way to obtain the SDR image.

[0302] It should be noted that the embodiments of the present application do not limit the format of SDR images. For example, in terms of color space, the SDR image can be a YUV image or an RGB image. In terms of the bit width of the data, the bit width of the SDR image can be 8bit, 10bit, 12bit, etc. The photoelectric conversion function of the SDR image can be a PQ function, a gamma function, a log function, an HLG function, etc.

[0303] S702, obtain a first image.

[0304] Exemplarily, when an SDR image is used as the basic image, an HDR image can be used as the first image.

[0305] Exemplarily, when an HDR image is used as the basic image, an SDR image can be used as the first image.

[0306] S703, obtain enhancement data and metadata of the enhancement data according to the basic image and the first image.

[0307] Exemplarily, S703 may include S7031 to S7034:

[0308] S7031, obtain a first intermediate image according to the basic image.

[0309] In a possible way, the basic image can be directly used as the first intermediate image.

[0310] In a possible way, the basic image can be encoded to obtain a bitstream of the basic image. Then, the bitstream of the basic image is decoded to obtain a reconstructed image of the basic image; the reconstructed image of the basic image is used as the first intermediate image. Among them, for the encoding and decoding of the basic image, codecs such as JPEG, HEIF, H.264, and HEVC can be used, and the embodiments of the present application do not limit this.

[0311] S7032, generate a second intermediate image according to the first intermediate image.

[0312] In a possible way, the first intermediate image data can be used as the second intermediate image.

[0313] In a possible way, a first local or global mapping relationship between the first intermediate image and the HDR image can be obtained; then, the first intermediate image can be mapped using this first mapping relationship to obtain the second intermediate image.

[0314] Exemplarily, multiple characteristic luminance values of the HDR image can be obtained; for example, the histogram of the HDR image is obtained, and the luminance at the peak position in the histogram is taken as the characteristic luminance value. As another example, the average luminance value of a specific region (such as a face, green plants, blue sky) in the HDR image is taken as the characteristic luminance value. Then, the pixel position of the characteristic luminance value in the HDR image is determined, and the luminance value of this pixel position in the basic image is determined. After that, according to the characteristic luminance value and the luminance value of this pixel position in the basic image, the first mapping relationship is determined.

[0315] Exemplarily, the first mapping relationship can include various forms. For example, forms such as Sigmoid, cubic spline, gamma, straight line, etc., or their inverse function forms are not limited in the embodiments of the present application. For example, the first mapping relationship can be represented by the following curve:

[0316]

[0317] Among them, L and L' in the formula can be normalized optical signals or electrical signals, a and b are scaling coefficients, n and m are power exponents, and p is a control parameter. The embodiments of the present application do not limit this.

[0318] As another example, the first mapping relationship can be the inverse curve of the above curve.

[0319] Exemplarily, the method of mapping the first intermediate image using the first mapping relationship (which can be represented by TMB1()) can be as follows: baseAfter[i] = TMB1(base[i]), or baseAfter[i] = TMB1() * base[i]. Where base[i] is the pixel value of the i-th pixel point in the first intermediate image, and baseAfter[i] is the pixel value of the i-th pixel point in the second intermediate image.

[0320] It should be noted that in the embodiments of the present application, baseAfter[i] and base[i] can be in any domain. baseAfter[i] and base[i] can be in a linear domain, PQ domain, or log domain, etc. And the present application also does not limit the color space of baseAfter[i] and base[i], which can be color spaces such as YUV, RGB, Lab, HSV, CIEXYZ, etc.

[0321] It should be noted that in one possible way, the first intermediate image can be subjected to color gamut mapping to convert the first intermediate image from the current color gamut to the target color gamut; then, the first intermediate image after color gamut mapping is mapped using the first mapping relationship to obtain a second intermediate image. In another possible way, after the second intermediate image is obtained by mapping the first intermediate image using the first mapping relationship, the second intermediate image can be subjected to color gamut mapping to convert the second intermediate image from the current color gamut to the target color gamut. Among them, the current color gamut and the target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0322] Exemplarily, the parameters of the first mapping relationship can be used as the metadata of the enhanced data. Among them, the parameters of the first mapping relationship can be the first data.

[0323] S7033, generate a third intermediate image according to the first image and the second intermediate image.

[0324] In one possible way, the third intermediate image = g(first image) / f(baseAfter[i]), where g() and f() are conversion functions in the numerical domain. For example, g() and f() can be log, the Optical-Electro Transfer Function (OETF), the Electro-Optical Transfer Function (EOTF), a linear function, a piecewise curve, etc. Exemplarily, g() and f() may also include processing of the second intermediate image; for example, obtaining the data range (such as the minimum value and / or the maximum value) of the second intermediate image, mapping the minimum value of the second intermediate image to 0 or a certain preset value, and / or mapping the maximum value of the second intermediate image to 1.0 or a certain preset value, and / or mapping the intermediate value of the second intermediate image to a certain intermediate value according to the second mapping relationship (the second mapping relationship can be determined according to the maximum value and / or the minimum value of the second intermediate image).

[0325] In one possible way, the third intermediate image = g(the first image) - f(baseAfter[i]), where g() and f() are conversion functions for the numerical range. For example, g() and f() can be log, OETF, EOTF, linear functions, piecewise curves, etc. Exemplarily, g() and f() may also include processing of the second intermediate image; for example, obtaining the data range (such as the minimum value and / or the maximum value) of the second intermediate image, mapping the minimum value of the second intermediate image to 0 or a certain preset value, and / or mapping the maximum value of the second intermediate image to 1.0 or a certain preset value, and / or mapping the intermediate value of the second intermediate image to a certain intermediate value according to the second mapping relationship (the second mapping relationship can be determined according to the maximum value and / or the minimum value of the second intermediate image).

[0326] Exemplarily, the parameters of the second mapping relationship can be used as the metadata of the enhanced data. Among them, the parameters of the second mapping relationship can be the first data.

[0327] S7034, process the third intermediate image to obtain the enhanced data and the metadata of the enhanced data.

[0328] Optionally, the third intermediate image can be downsampled to obtain a fourth intermediate image (which can be represented by enchence).

[0329] In one possible way, the data range (such as the minimum value, the maximum value, etc.) of the fourth intermediate image can be obtained, the minimum value of the fourth intermediate image can be mapped to 0 or a certain preset value, and / or the maximum value of the fourth intermediate image can be mapped to 1.0 or a certain preset value; and / or the intermediate value of the fourth intermediate image can be mapped to a certain intermediate value according to the third mapping relationship (the third mapping relationship can be determined according to the maximum value and / or the minimum value of the fourth intermediate image) to obtain the enhanced data. Among them, the maximum value, the minimum value, and the intermediate value of the fourth intermediate image can all be used as metadata; among them, the maximum value, the minimum value, and the intermediate value of the fourth intermediate image can be the first data.

[0330] Exemplarily, the parameters of the third mapping relationship can be used as the metadata of the enhanced data. Among them, the parameters of the third mapping relationship can be the first data.

[0331] In a possible way, a histogram of the fourth intermediate data can be obtained, and the histogram equalization method can be used to process the histogram of the fourth intermediate data to obtain the fourth mapping relationship TMB2(). Then, the fourth intermediate image can be mapped according to the fourth mapping relationship to obtain enhanced data. For example, the fourth intermediate image can be mapped in the following way: enchenceAfter[i] = TMB2(enchence[i]), or, enchenceAfter[i] = TMB2() * enchence[i]. enchenceAfter[i] is the enhanced data corresponding to the i-th pixel in the basic image, and enchence[i] is the i-th pixel in the fourth intermediate image.

[0332] Exemplarily, the parameters of the fourth mapping relationship can be used as the metadata of the enhanced data. Among them, the parameters of the fourth mapping relationship can be the first data.

[0333] That is to say, the metadata of the enhanced data can include the data that affects the processing result in the process of determining the enhanced data based on the basic image and the first image.

[0334] It should be understood that the embodiments of the present application do not limit the manner of determining the enhanced data and the metadata of the enhanced data based on the basic image and the first image.

[0335] Exemplarily, the fourth mapping relationship has various forms, such as Sigmoid, cubic spline, gamma, straight line, linear, piecewise curve, etc. forms or the forms of their inverse functions. The embodiments of the present application do not limit this.

[0336] It should be noted that in the embodiments of the present application, enchenceAfter[i] and enchence[i] can be in any domain; for example, enchenceAfter[i] and enchence[i] can be in a linear domain, a PQ domain, or a log domain, etc. The present application also does not limit the color space of enchenceAfter[i] and enchence[i], which can be color spaces such as YUV, RGB, Lab, HSV, etc.

[0337] It should be noted that, in one possible way, color gamut mapping can be performed on the fourth intermediate image to convert the fourth intermediate image from the current color gamut to the target color gamut; then, the fourth mapping relationship is used to map the fourth intermediate image after color gamut mapping to obtain enhanced data. In another possible way, after using the fourth mapping relationship to map the fourth intermediate image to obtain enhanced data, color gamut mapping can be performed on the enhanced data to convert the enhanced data from the current color gamut to the target color gamut. Among them, the current color gamut and the target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0338] S704, encode the base image.

[0339] Exemplarily, the process of encoding the base image can refer to the image encoding process in the prior art and will not be elaborated here.

[0340] S705, encode the enhanced data.

[0341] Exemplarily, one pixel in the base image can correspond to one enhanced data, or multiple pixels can correspond to one enhanced data. Among them, the enhanced data may contain multiple values, and the number of values contained in the enhanced data corresponding to each pixel is the same.

[0342] Exemplarily, the enhanced data can be encapsulated into the SEI of HEVC or VVC, the user-defined NAL unit, or the reserved NAL unit, or it can also be the APP extension information encapsulated by JFIF, or the data segment encapsulated by MP4, etc.

[0343] Exemplarily, the enhanced data can also be encapsulated into the file location indicated by the metadata location information, such as the location after the EOI (end of image) of the complete JPEG file.

[0344] It should be understood that the embodiments of the present application do not limit the position of the enhanced data in the bitstream.

[0345] Exemplarily, the metadata of the enhanced data can be encoded with reference to S706 - S711 as follows.

[0346] S706, obtain the floating-point data format

[0347] S707, obtain the bit width of the second data.

[0348] S708, process the first data according to the floating-point data format and the bit width to obtain the second data.

[0349] S709, write the floating-point data format into the bitstream.

[0350] S710, write the bit width of the second data into the bitstream.

[0351] S711, Write the second data into the bitstream.

[0352] Exemplarily, S706 to S711 may refer to the description of S402 to S407, which will not be elaborated here.

[0353] It should be noted that the embodiments of the present application do not limit the encoders for encoding the base image, enhancement data, and metadata of the enhancement data. For example, JPEG, HEIF, H.264, HEVC, etc.

[0354] It should be noted that the embodiments of the present application do not limit the execution order of encoding the base image, enhancement data, and metadata of the enhancement data.

[0355] It should be noted that the encoded base image, encoded enhancement data, and encoded metadata of the enhancement data can be stored in one bitstream / file. The encoded base image, encoded enhancement data, and encoded metadata of the enhancement data can be regarded as three parts of one bitstream / file.

[0356] Figure 8 Schematic diagram of the decoding process 800 for the exemplary image and metadata of the image. Figure 8 The decoding process 800 in Figure 7 corresponds to the encoding process 700 of

[0357] S801, Obtain the reconstructed image of the base image, the reconstruction result of the enhancement data, and the encoded metadata from the received bitstream.

[0358] Among them, S801 may refer to the description of S902 to S905 below, which will not be elaborated here.

[0359] Exemplarily, the encoded metadata may include the second data.

[0360] S802, Process the encoded metadata according to the floating-point data format to obtain the reconstructed metadata.

[0361] Exemplarily, for the second data, the second data may be processed according to the floating-point data format to obtain the reconstructed second data, that is, the first data.

[0362] S803, Merge the reconstructed image of the base image and the reconstruction result of the enhancement data according to the reconstructed metadata to obtain the reconstructed first image.

[0363] Exemplarily, when the base image is an HDR image and the first image is an SDR image, if the decoding end only supports SDR display, the reconstructed SDR image can be obtained by merging the reconstructed image of the base image and the reconstruction result of the enhancement data according to the reconstructed metadata; if the decoding end supports HDR image display, the reconstructed image of the base image can be directly displayed.

[0364] Exemplarily, when the base image is an SDR image and the first image is an HDR image, if the decoding end supports HDR display, the reconstructed HDR image can be obtained by merging the reconstructed image of the base image and the reconstruction result of the enhancement data according to the reconstructed metadata; if the decoding end only supports SDR image display, the reconstructed image of the base image can be directly displayed.

[0365] Figure 9 Schematic diagram of the decoding process 900 of the exemplary image and the metadata of the image. Figure 9 The decoding process 900 in Figure 8 is a specific description of the decoding process 800 in

[0366] S901, Receive the bitstream.

[0367] S902, Decode the reconstructed image of the base image from the bitstream.

[0368] Exemplarily, any decoder (such as HEVC, JPEG, ProRes, HEIF) can be used to decode the received bitstream to obtain the reconstructed image of the base image.

[0369] It should be noted that the embodiments of the present application do not limit the format of the reconstructed image of the base image. In terms of color space, the reconstructed image of the base image can be a YUV image or an RGB image. In terms of the bit width of the data, the bit width of the reconstructed image of the base image can be 8bit, 10bit, 12bit, etc.

[0370] S903, Decode the reconstruction result of the enhancement data from the bitstream.

[0371] Exemplarily, the reconstruction result of the enhancement data can be decoded from the SEI of HEVC or VVC, the user-defined NAL unit or the reserved NAL unit, the APP extension information encapsulated in JFIF, or the data segment encapsulated in MP4.

[0372] Exemplarily, the reconstruction result of the enhancement data can also be decoded from the file location indicated by the metadata location information, such as the location after the EOI (end of image) of the complete JPEG file.

[0373] It should be understood that the reconstruction result of the enhanced data can also be decoded from other file locations, and the embodiments of the present application do not limit this.

[0374] S904, Parse the second data, the floating-point data format, and the bit width of the second data from the bitstream.

[0375] S905, Process the second data according to the floating-point data format and the bit width of the second data to obtain the first data.

[0376] Exemplarily, S904~S905 can refer to the description of the above S502~S503 and will not be elaborated here.

[0377] Exemplarily, the embodiments of the present application do not limit the execution order of S902, S903, and S904.

[0378] It should be understood that other reconstruction data (the other reconstruction data is the reconstruction result of the data other than the first data in the metadata of the enhanced data) can also be decoded from the bitstream, and then the reconstruction metadata of the enhanced data can be obtained.

[0379] S906, According to the reconstruction metadata of the enhanced data, merge the reconstructed image of the basic image and the reconstruction result of the enhanced data to obtain the first reconstructed image.

[0380] Exemplarily, according to the reconstruction metadata of the enhanced data, the reconstructed image of the basic image and the reconstruction result of the enhanced data can be merged to obtain an SDR reconstructed image or an HDR reconstructed image; the present application takes the basic image as an SDR image and the first image as an HDR image as an example for illustration; it may include the following S1~S6:

[0381] S1, Process the reconstructed image of the basic image according to the reconstruction metadata of the enhanced data to obtain the first reconstructed image.

[0382] In a possible way, take the reconstructed image of the basic image as the first reconstructed image, that is, R_baseAfter[i]=R_Base[i]. Wherein, R_baseAfter[i] represents the pixel value of the i-th pixel point in the first reconstructed image, and R_Base[i] represents the pixel value of the i-th pixel point in the reconstructed image of the basic image.

[0383] In a possible way, obtain the lower limit THL of the basic image from the reconstruction metadata of the enhanced data, and obtain the first reconstructed image according to the following method: R_baseAfter[i]=R_Base[i]+THL.

[0384] In one possible way, parameters of a first mapping relationship and a lower limit THL of a basic image are obtained from reconstruction metadata of enhanced data; the first mapping relationship TMB1() is obtained according to the parameters of the first mapping relationship. Then, a first reconstructed image is obtained according to the following method: R_baseAfter[i] = TMB1(R_Base[i] + THL).

[0385] In one possible way, parameters of a first mapping relationship are obtained from reconstruction metadata of enhanced data, and the first mapping relationships: TMB11() and TMB12() are obtained according to the parameters; where TMB11() can be a global and / or local tone mapping function, and TMB12 can be in forms such as linear, spline, piecewise curve, etc. Then, a first reconstructed image is obtained according to the following method: R_baseAfter[i] = TMB11(TMB12(R_Base[i])).

[0386] It should be noted that in the embodiments of the present application, R_baseAfter[i] and R_Base[i] can be in any domain; for example, R_baseAfter[i] and R_Base[i] can be in a linear domain, a PQ domain, or a log domain, etc. And the present application does not limit the color space of R_baseAfter[i] and R_Base[i], which can be color spaces such as YUV, RGB, Lab, HSV, etc.

[0387] It should be noted that in one possible way, a gamut mapping can be performed on the reconstructed image of the basic image to convert the reconstructed image of the basic image from the current gamut to the target gamut; then, according to the reconstruction metadata of the enhanced data, the reconstructed image of the basic image after gamut mapping is processed to obtain a first reconstructed image. In another possible way, after processing the reconstructed image of the basic image according to the reconstruction metadata of the enhanced data to obtain a first reconstructed image, a gamut mapping can be performed on the first reconstructed image to convert the first reconstructed image from the current gamut to the target gamut. Among them, the current gamut and the target gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0388] S2. The reconstruction result of the enhanced data is subjected to a transformation process according to the reconstruction metadata of the enhanced data to obtain a first reconstruction result.

[0389] Exemplarily, the transformation process can be to upsample the reconstruction result of the enhanced data to obtain a first reconstruction result. For example, a certain preset filter can be used for upsampling, or an upsampling method including multiple directional interpolations can be adopted, or bicubic spline method can be used for upsampling. The embodiments of the present application do not limit this.

[0390] S2. Process the first reconstruction result according to the reconstruction metadata of the enhanced data to obtain a second reconstruction result.

[0391] Optionally, the first reconstruction result (which can be referred to as R_enchence) can be normalized. Usually, the range of 8-bit data is from 0 to 255. By dividing all numbers in R_enchence by 255.0, it is converted to the range from 0 to 1.0. The embodiments of the present application do not limit this.

[0392] In a possible way, the first reconstruction result or the normalized first reconstruction result is used as the second reconstruction result, that is, R_enchenceAfter[i] = R_enchence[i]. R_enchenceAfter[i] is the i-th pixel point of the second reconstruction result, and R_enchence[i] is the i-th pixel point of the first reconstruction result / or the normalized first reconstruction result.

[0393] In a possible way, the parameters of the fourth mapping relationship, the upper limit THL and the lower limit THH of the enhanced data can be obtained from the reconstruction metadata of the enhanced data. According to the parameters of the fourth mapping relationship, the fourth mapping relationship TMB2() is obtained, and R_enchenceAfter[i] = TMB2(R_enchence[i] * THH + (A - R_enchence[i]) * THL).

[0394] In a possible way, the parameters of the fourth mapping relationship, the upper limit THL and the lower limit THH of the enhanced data can be obtained from the reconstruction metadata of the enhanced data. According to the parameters of the fourth mapping relationship, the fourth mapping relationship TMB2() is obtained, and R_enchenceAfter[i] = TMB2(R_enchence[i]) * THH + (A - TMB2(R_enchence[i])) * THL.

[0395] It should be noted that in the embodiments of the present application, R_enchenceAfter[i] and R_enchence[i] can be in any domain. For example, R_enchenceAfter[i] and R_enchence[i] can be in a linear domain, a PQ domain, a log domain, etc. The present application also does not limit the color space of R_enchenceAfter[i] and R_enchence[i], which can be color spaces such as YUV, RGB, Lab, HSV, etc.

[0396] It should be noted that in one possible way, the first reconstruction result can be subjected to color gamut mapping to convert the first reconstruction result from the current color gamut to the target color gamut; then, the first reconstruction result after color gamut mapping is mapped using the fourth mapping relationship to obtain the second reconstruction result. In another possible way, after the second reconstruction result is obtained by mapping the first reconstruction result using the fourth mapping relationship, the second reconstruction result can be subjected to color gamut mapping to convert the second reconstruction result from the current color gamut to the target color gamut. Among them, the current color gamut and the target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0397] After that, the second reconstructed image recHDR can be obtained according to R_baseAfter and R_enhenceAfter: recHDR[i] = g(baseAfter[i]) * f(R_enchenceAfter[i]) or recHDR[i] = R_baseAfter[i] + f(R_enchenceAfter[i]), where g() and f() are conversion functions in the numerical domain. recHDR[i] can be any component of the RGB or YUV components, and f(R_enchenceAfter[i]) is the gain of any component obtained from the reconstruction result of the enhanced data.

[0398] It should be noted that the present invention does not limit whether other image processing is performed on the second reconstructed image recHDR before display after it is obtained.

[0399] S4. Generate the HDR reconstructed image recHDRAfter according to the reconstruction metadata of the enhanced data.

[0400] In one possible way, the second reconstructed image can be used as the HDR reconstructed image, that is, recHDRAfter[i] = recHDR[i].

[0401] In one possible way, the lower limit THL of the second reconstructed image can be obtained from the reconstruction metadata of the enhanced data, and the way to generate the HDR reconstructed image can be as follows: recHDRAfter[i] = recHDR[i] + THL.

[0402] In one possible way, the upper limit THH of the second reconstructed image can be obtained from the reconstruction metadata of the enhanced data, and the way to generate the HDR reconstructed image can be as follows: recHDRAfter[i] = TMB1(R_base[i]).

[0403] It should be noted that the present invention does not limit the forms of f(x) and TMB1(), that is, recHDRAfter[i] and recHDR[i] can be any domain. For example, recHDRAfter[i] and recHDR[i] can be a linear domain, or a PQ domain, a log domain, etc. The present invention also does not limit the color space of recHDRAfter[i] and recHDR[i], which can be color spaces such as YUV, RGB, Lab, HSV, etc.

[0404] Since the encoding and decoding method of the reconstructed metadata of the enhanced data in the embodiments of the present application will not cause loss of precision of the reconstructed metadata of the enhanced data, and the reconstructed metadata of the enhanced data obtained at the decoding end is the same as the metadata of the enhanced data encoded at the encoding end, the quality of the SDR image or HDR image obtained at the decoding end can be guaranteed.

[0405] Figure 10 The figure is a schematic diagram of the exemplary encoding device 1000. The encoding device 1000 can be used to execute the method of the foregoing embodiments. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0406] Refer to Figure 10 , exemplarily, the encoding device 1000 may include:

[0407] A metadata acquisition module 1001, configured to acquire metadata of multimedia; wherein, the metadata of multimedia includes first data, and the first data is floating-point data;

[0408] A data writing module 1002, configured to write second data into a bitstream; wherein, the second data is a representation form of the first data in a floating-point data format.

[0409] Exemplarily, the encoding device 1000 further includes:

[0410] A first data generation module, configured to generate second data based on the first data.

[0411] Exemplarily, the encoding device 1000 further includes:

[0412] A first format acquisition module, configured to acquire a floating-point data format;

[0413] The first data generation module is specifically configured to process the first data according to the floating-point data format to obtain the second data.

[0414] Exemplarily, the encoding device 1000 further includes:

[0415] A first bit width acquisition module, configured to acquire the bit width of the second data;

[0416] The first data generation module is specifically configured to process the first data according to the floating-point data format and the bit width of the second data to obtain the second data.

[0417] Exemplarily, the data writing module 1002 is further configured to write the bit width into the bitstream.

[0418] Exemplarily, the data writing module 1002 is further configured to write the floating-point data format into the bitstream.

[0419] Exemplarily, the second data is the binary representation or hexadecimal representation of the first data in the floating-point data format.

[0420] Exemplarily, the metadata of the multimedia is the metadata of a high dynamic range (HDR) image.

[0421] Exemplarily, the metadata of the multimedia is the metadata of enhanced data, and the enhanced data is generated based on a standard dynamic range (SDR) image and an HDR image.

[0422] Exemplarily, the floating-point data format includes at least one of the following: IEEE 754, FP8 (E4M3), FP8 (E5M2), Bfloat16, TensorFloat-32.

[0423] Figure 11 It is a schematic diagram of the exemplary decoding device 1100 shown. The decoding device 1100 can be used to execute the method of the foregoing embodiments. Therefore, the beneficial effects it can achieve can refer to the beneficial effects in the corresponding method provided above, which will not be elaborated here.

[0424] Refer to Figure 11 , exemplarily, the decoding device 1100 may include:

[0425] The bitstream receiving module 1101 is configured to receive the bitstream;

[0426] The parsing module 1102 is configured to parse the second data from the bitstream, where the second data is the representation of the first data in the floating-point data format, and the first data is the floating-point data in the metadata of the multimedia;

[0427] The second data generation module 1103 is configured to generate the first data based on the second data.

[0428] Exemplarily, the decoding device 1100 further includes:

[0429] The second format acquisition module is configured to acquire the floating-point data format;

[0430] The second data generation module 1103 is specifically configured to process the second data according to the floating-point data format to obtain the first data.

[0431] Exemplarily, the second format acquisition module includes: a parsing module 1102; the parsing module 1102 is further configured to parse floating-point data format from the bitstream.

[0432] Exemplarily, the second bitwidth acquisition module is configured to acquire the bitwidth of the second data;

[0433] The second data generation module 1103 is specifically configured to process the second data according to the floating-point data format and the bitwidth to obtain the first data.

[0434] Exemplarily, the second bitwidth acquisition module includes: a parsing module 1102; the parsing module 1102 is further configured to parse the bitwidth from the bitstream.

[0435] Exemplarily, the second data is a binary representation or a hexadecimal representation of the first data in floating-point data format.

[0436] Exemplarily, the metadata of the multimedia is the metadata of a high dynamic range (HDR) image.

[0437] Exemplarily, the metadata of the multimedia is the metadata of enhanced data, and the enhanced data is determined based on a standard dynamic range (SDR) image and a high dynamic range (HDR) image.

[0438] Exemplarily, the floating-point data format includes at least one of the following: IEEE 754, FP8 (E4M3), FP8 (E5M2), Bfloat16, TensorFloat-32.

[0439] The following Figure 12 describes the codec system to which the present application is applied. Figure 12 FIG. is a schematic block diagram of a codec system applied to an embodiment of the present application. For example, a video codec system 10 (or simply referred to as a codec system 10) that can utilize the technology of the present application can be used. The video encoder 20 (or simply referred to as an encoder 20) and the video decoder 30 (or simply referred to as a decoder 30) in the video codec system 10 represent devices that can be used to execute various techniques according to the various examples described in the present application.

[0440] As Figure 12 shown, the codec system 10 includes a source device 12, and the source device 12 is configured to provide encoded data 21 such as an encoded image to a destination device 14 for decoding the encoded data.

[0441] The source device 12 includes an encoder 20. Optionally, it may further include an image source 16, an image preprocessor 18 (or a preprocessing unit), a communication interface, or a communication unit 22.

[0442] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images)). The image source may be any type of memory or storage that stores any of the above images.

[0443] To distinguish the processing performed by the preprocessor 18 or the preprocessing unit 18, the image or image data 17 may also be referred to as the original image or original image data 17.

[0444] The preprocessor 18 is configured to receive (original) image data 17 and preprocess the image data 17 to obtain preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or denoising. It can be understood that the preprocessing unit 18 may be an optional component.

[0445] The video encoder 20 is configured to receive the preprocessed image data 19 and provide encoded image data 21.

[0446] The communication interface 22 in the source device 12 may be used to: receive the encoded image data 21 and transmit the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0447] The destination device 14 includes a decoder 30 (e.g., a video decoder 30), and optionally, may include a communication interface or communication unit 28, a postprocessor 32 (or postprocessing unit 32), and a display device 34.

[0448] The communication interface 28 in the destination device 14 is configured to directly receive the encoded image data 21 (or any other processed version) from the source device 12 or from any other source device such as a storage device. For example, the storage device is an encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0449] The communication interfaces 22 and 28 can be used to send or receive the encoded image data 21 or the encoded data through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, etc., or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination of any type thereof.

[0450] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a packet, and / or use any type of transport encoding or processing to process the encoded image data for transmission over the communication link or communication network.

[0451] The communication interface 28 corresponds to the communication interface 22. For example, it can be used to receive the transmitted data and process the transmitted data using any type of corresponding transport decoding or processing and / or de-encapsulation to obtain the encoded image data 21.

[0452] Both the communication interface 22 and the communication interface 28 can be configured as Figure 12 a unidirectional communication interface as indicated by the arrow of the corresponding communication channel 13 pointing from the source device 12 to the destination device 14 in

[0453] The decoder 30 is used to receive the encoded image data 21 and provide the decoded image data 31 or the decoded image 31.

[0454] The post-processor 32 in the destination device 14 is used to post-process the decoded image data 31 such as the decoded image 31 (also referred to as the reconstructed image data) to obtain the post-processed image data 33 such as the post-processed image 33. The post-processing performed by the post-processing unit 32 can include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.

[0455] The display device 34 in the destination device 14 is used to receive the post-processed image data 33 to display an image to a user, viewer, etc. The display device 34 can be or include any type of display for representing the reconstructed image. For example, an integrated or external display screen or monitor. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0456] Although Figure 12 the source device 12 and the destination device 14 are shown as separate devices, the device embodiments can also include both the source device 12 and the destination device 14 or the functions of both the source device 12 and the destination device 14 simultaneously, that is, include both the source device 12 or the corresponding function and the destination device 14 or the corresponding function simultaneously. In these embodiments, the source device 12 or the corresponding function and the destination device 14 or the corresponding function can be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0457] According to the description, Figure 12 the presence and (accurate) division of different units or functions in the shown source device 12 and / or destination device 14 may vary according to the actual device and application, which is obvious to those skilled in the art.

[0458] Next, in combination with Figure 13A describe the content providing system of the content distribution service to which this application is applied, Figure 13A is a block diagram of a content providing system for implementing a content distribution service applied to an embodiment of this application. The content providing system 2100 includes a capture device 2102, a terminal device 2106, and (optionally) a display 2126. The capture device 2102 communicates with the terminal device 2106 through a communication link 2104. The communication link can include the above-mentioned communication channel 13. The communication link 2104 includes but is not limited to WiFi, Ethernet, wired, wireless (3G / 4G / 5G), USB, or any combination of these types.

[0459] The capture device 2102 generates data and can encode the data using the encoding method shown in the above embodiments. Alternatively, the capture device 2102 can distribute the data to a streaming media server (not shown in the figure), and the server encodes the data and transmits the encoded data to the terminal device 2106. The capture device 2102 includes, but is not limited to, a camera, a smartphone or a tablet computer, a computer or a laptop, a video conferencing system, a PDA, an in-vehicle device, or any combination thereof. For example, the capture device 2102 can include the above-mentioned source device 12. When the data includes video, the video encoder 20 in the capture device 2102 can actually perform video encoding processing. When the data includes audio (i.e., sound), the audio encoder 20 in the capture device 2102 can actually perform audio encoding processing. In some actual scenarios, the capture device 2102 distributes the encoded video data and the encoded audio data by multiplexing them together. In other actual scenarios, such as in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. The capture device 2102 distributes the encoded audio data and the encoded video data to the terminal device 2106 separately.

[0460] The terminal device 2106 in the content providing system 2100 receives and regenerates the encoded data. The terminal device 2106 can be a device with data receiving and restoring capabilities, such as a smartphone or a tablet computer 2108, a computer or a laptop 2110, a network video recorder (NVR) / digital video recorder (DVR) 2112, a television 2114, a set top box (STB) 2116, a video conferencing system 2118, a personal digital assistant (PDA) 2122, an in-vehicle device 2124, or any combination thereof, or such a device capable of decoding the above-mentioned encoded data. For example, the terminal device 2106 can include the above-mentioned destination device 14. When the encoded data includes video, the video decoder 30 in the terminal device preferentially performs video decoding. When the encoded data includes audio, the audio decoder in the terminal device preferentially performs audio decoding processing. The terminal device 2106 can be a video playback application, a streaming media playback application, a streaming media playback platform, or a live broadcast platform running on the terminal device.

[0461] For a terminal device with a display, such as a smartphone or a tablet 2108, a computer or a laptop 2110, an NVR / DVR 2112, a TV 2114, a PDA 2122, or a vehicle-mounted device 2124, the terminal device can send the decoded data to its display. For a terminal device without a display, such as an STB 2116 or a video conferencing system 3118, the device is connected to an external display 2126 to receive and display the decoded data.

[0462] When each device in the system performs encoding or decoding, the image encoding device or the image decoding device shown in the above embodiments can be used.

[0463] Figure 13B For Figure 13A FIG. is a schematic diagram of an exemplary structure of the terminal device 2106. After the terminal device 2106 receives a bitstream from the capture device 2102, the protocol processing unit 2202 analyzes the transport protocol of the bitstream. The protocols include, but are not limited to, the Real Time Streaming Protocol (RTSP), the Hyper Text Transfer Protocol (HTTP), the HTTP Live streaming protocol (HLS), the MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH), the Real-time Transport protocol (RTP), the Real Time Messaging Protocol (RTMP), or any combination thereof.

[0464] After the protocol processing unit 2202 processes the stream, a stream file is generated. The file is output to the demultiplexing unit 2204. The demultiplexing unit 2204 can separate the multiplexed data into encoded audio data and encoded video data. As described above, in other actual scenarios, such as in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. In this case, the encoded data is not transmitted to the video decoder 3206 and the audio decoder 2208 through the demultiplexing unit 2204.

[0465] Through demultiplexing processing, an elementary stream (ES) of video, an audio ES, and optional subtitles are generated. The video decoder 2206, including the video decoder 30 as described in the above embodiments, decodes the video ES by the decoding method as shown in the above embodiments to generate video frames, and sends the data to the synchronization unit 2212. The audio decoder 2208 decodes the audio ES to generate audio frames, and sends the data to the synchronization unit 2212. Alternatively, the video frames may be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212. Similarly, the audio frames may be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212.

[0466] The synchronization unit 2212 synchronizes the video frames and audio frames, and provides the video / audio to the video / audio display 3214. For example, the synchronization unit 2212 synchronizes the presentation of video and audio information. The information may be sent with relevant timestamps using encoded audio and visual data, and the relevant timestamps are encoded in the syntax.

[0467] If the bitstream includes subtitles, the subtitle decoder 2210 decodes the subtitles, synchronizes them with the video frames and audio frames, and provides the video / audio / subtitle to the video / audio / subtitle display 2216.

[0468] The present invention is not limited to the above system, and the image encoding device or image decoding device in the above embodiments can be used in other systems such as automobiles.

[0469] Next, in conjunction with Figure 14A describe the streaming media system applicable to the embodiments of the present application. Figure 14A It is a schematic diagram of the working process of a streaming media system applicable to the embodiments of the present application.

[0470] The streaming media system includes a content creation module that generates the required content data, such as video or audio. The streaming media system also includes a video encoding module that encodes the generated content through an encoder. The streaming media system also includes a video stream transmission module that transmits the encoded video in the form of a bitstream. Optionally, the format of the video stream can be converted into a bitstream format of a transmission protocol commonly used by OTT (over-the-top) devices. For example, the protocols include, but are not limited to, Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport protocol (RTP), Real Time Messaging Protocol (RTMP), or any combination thereof. Optionally, video stream storage can be performed to store the original format of the video stream and / or multiple converted bitstream formats for easy use. Further, the streaming media system also includes a video stream encapsulation module for encapsulating the video stream to generate an encapsulated video stream, which can be referred to as a video streaming media packet. Exemplarily, a video streaming media packet can be generated based on the transcoded video stream or the stored video stream. Further, the streaming media system also includes a content distribution network (CDN), and the CDN is used to distribute the video streaming media packet to multiple OTT devices, such as mobile phones, computers, tablets, home projectors, etc.

[0471] It should be noted that video encoding, video stream transmission, video stream transcoding, video stream storage, video streaming media packet generation, and content distribution network can all be implemented on a cloud server.

[0472] The following combines Figure 14B to describe an exemplary streaming media system architecture of the present application. This streaming media system architecture includes: a client device, a content distribution network, and a cloud server.

[0473] A user on the client device sends a play or playback request to the cloud platform. Optionally, the content of the request can be the title of a movie or TV program to be played.

[0474] The cloud platform makes a decision, replies to the client, and sends the address of the client request content on the CDN to the client. Optionally, the content sent to the client can be a URL link (uniform resource locator). Specifically, the playback application service in the cloud platform checks user authorization and permissions, and then considers various client characteristics and the current network conditions to determine which specific files are needed to process the playback request. It should be noted that the Content Delivery Network (CDN) regularly reports its operating status, learned routes, and available content (files) to the cache control service in the cloud platform.

[0475] Then the client requests the playback content from the CDN according to the address, and the CDN provides the content to the client, finally completing the client's request.

[0476] The following combines Figure 15A to describe the system architecture applicable to the embodiments of this application. Figure 15A is a schematic diagram of a possible system architecture applicable to the embodiments of this application. The system architecture of the embodiments of this application includes: a front-end device, a transmission link, and a terminal display device.

[0477] Among them, the front-end device is used to collect or produce HDR / SDR content (for example, HDR / SDR video or image).

[0478] In a possible embodiment, the front-end device can also be used to extract corresponding metadata from the HDR content. The metadata can include global mapping information, local mapping information, and dynamic metadata and static metadata corresponding to the HDR content. The front-end device can send the HDR content and metadata to the terminal display device through the transmission link. Specifically, the HDR content and metadata can be transmitted in the form of a single data packet, or separately in two data packets. The embodiments of this application do not make specific limitations.

[0479] Optionally, the terminal display device can be used to receive the metadata and HDR content, and obtain the mapping curve according to the global mapping information, local mapping information, and the terminal display device information included in the corresponding metadata extracted from the HDR content, perform global tone mapping and local tone mapping on the HDR content, convert it into display content adapted to the HDR display device or SDR device in the terminal display device, and display it. It should be understood that in different embodiments, the terminal display device can include a display device with a display capability having a lower dynamic range or a higher dynamic range than the HDR content generated by the front-end device. This application does not make a limitation.

[0480] Optionally, the front-end device and the terminal display device in the present application can be independent and different physical devices. For example, the front-end device can be a video capture device or a video production device. The video capture device can be a camera, a camera, an image drawing machine, etc. The terminal display device can be a device with a video playback function such as a virtual reality (VR) glasses, a mobile phone, a tablet, a TV, a projector, etc.

[0481] Optionally, the transmission link between the front-end device and the terminal display device can be a wireless connection or a wired connection. The wireless connection can adopt technologies including, for example, long term evolution (LTE), 5th generation (5G) mobile communication, and future mobile communications. The wireless connection can also include technologies such as wireless-fidelity (WiFi), Bluetooth, Near Field Communication (NFC), etc. The wired connection can include Ethernet connection, local area network connection, etc. There is no specific limitation on this.

[0482] The present application can also integrate the functions of the front-end device and the terminal display device on the same physical device. For example, terminal devices such as mobile phones and tablets with video shooting functions. The present application can also integrate some functions of the front-end device and some functions of the terminal display device on the same physical device. There is no specific limitation on this.

[0483] The following combines Figure 15B to describe the end-to-end image processing system provided by the embodiments of the present application. This system can be applied to Figure 15A the system architecture shown. Figure 15B FIG. is a schematic structural diagram of an image processing system provided by an embodiment of the present application. In Figure 14B , by way of example, HDR / SDR content takes HDR video as an example. The image processing system includes: an HDR preprocessing module, an HDR video encoding module, an HDR video decoding module, and a tone mapping module.

[0484] Among them, the HDR preprocessing module and the HDR video encoding module can be located in Figure 15A the front-end device shown, and the HDR video decoding module and the tone mapping module can be located in Figure 15A the terminal display device shown.

[0485] HDR Preprocessing Module: It is used to extract dynamic metadata from an HDR video (e.g., the maximum, minimum, average, and variation range of brightness), and determine mapping curve parameters according to the dynamic metadata and the display capabilities of the target display device. Then, write the mapping curve parameters into the dynamic metadata to obtain HDR metadata, and transmit it. Among them, the HDR video can be captured or an HDR video processed by a colorist; the display capabilities of the target display device refer to the brightness range that the target display device can display.

[0486] HDR Video Encoding Module: It is used to perform video encoding on the HDR video and HDR metadata according to a video compression standard (e.g., AVS or HEVC standard) (e.g., embed the HDR metadata into a user-defined part of the bitstream), and output the corresponding bitstream (AVS or HEVC bitstream).

[0487] HDR Video Decoding Module: It is used to decode the generated bitstream (AVS bitstream or HEVC bitstream) according to the standard corresponding to the bitstream format, and output the decoded HDR video and HDR metadata.

[0488] Tone Mapping Module: It is used to generate a mapping curve according to the parameters of the mapping curve in the decoded HDR metadata, and perform tone mapping (i.e., HDR adaptation processing or SDR adaptation processing) on the decoded HDR video, and send the HDR-adapted video after tone mapping to the HDR display terminal for display, or send the SDR-adapted video to the SDR display terminal for display.

[0489] Exemplarily, the HDR preprocessing module can be present in a video capture device or a video production device.

[0490] Exemplarily, the HDR video encoding module can be present in a video capture device or a video production device.

[0491] Exemplarily, the HDR video decoding module can be present in a set-top box, a television display device, a mobile terminal display device, and video conversion devices such as network live broadcasts and network video applications.

[0492] Exemplarily, the tone mapping module can be present in a set-top box, a television display device, a mobile terminal display device, and video conversion devices such as network live broadcasts and network video applications. More specifically, the tone mapping module can exist in the form of a chip or a software program in a set-top box, a television display, and a mobile terminal display, and can exist in the form of a software program in video conversion devices such as network live broadcasts and network video applications.

[0493] In a possible embodiment, when both the tone mapping module and the HDR video decoding module are present in the set-top box, the set-top box can complete the functions of receiving, decoding, and tone mapping of the video bitstream. The set-top box sends the decoded video data to the display device for display through a high definition multimedia interface (HDMI), enabling the user to enjoy the video content.

[0494] In one example, Figure 16 FIG. shows a schematic block diagram of a device 1600 according to an embodiment of the present application. The device 1600 may include: a processor 1601 and a transceiver / transceiver pin 1602. Optionally, it may further include a memory 1603.

[0495] Each component of the device 1600 is coupled together through a bus 1604. The bus 1604 includes, in addition to the data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all various buses are referred to as bus 1604 in the figure.

[0496] Optionally, the memory 1603 may be used to store the instructions in the foregoing method embodiments. The processor 1601 may be used to execute the instructions in the memory 1603, control the receiving pin to receive signals, and control the sending pin to send signals.

[0497] The device 1600 may be the electronic device or the chip of the electronic device in the foregoing method embodiments.

[0498] Exemplarily, the electronic device in the foregoing embodiments may be a terminal device or a server.

[0499] All relevant contents of each step involved in the foregoing method embodiments may be cited in the function descriptions of the corresponding functional modules and will not be elaborated herein.

[0500] The embodiment of the present application further provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits. When the one or more processors execute computer instructions, the steps of the relevant method described above are executed to implement the method in the foregoing embodiments. Among them, the interface circuit is the transceiver / transceiver pin 1602.

[0501] The present embodiment further provides a computer-readable storage medium. Computer instructions are stored in the computer-readable storage medium. When the computer instructions run on the electronic device, the electronic device is caused to execute the relevant method steps to implement the method in the foregoing embodiments.

[0502] This embodiment also provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer is enabled to execute the above-related steps to implement the method in the above embodiment.

[0503] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected thereto. Among them, the memory is used to store computer execution instructions. When the device runs, the processor may execute the computer execution instructions stored in the memory, so that the chip executes the methods in the above method embodiments.

[0504] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0505] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0506] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0507] The unit described as a separated component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0508] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0509] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0510] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs, etc., which can store program codes.

[0511] The steps of the methods or algorithms described in combination with the disclosed content of the embodiments of the present application can be implemented in a hardware manner or by a processor executing software instructions. The software instructions can be composed of corresponding software modules. The software modules can be stored in a random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, compact disc read-only memories (CD-ROMs), or any other form of storage medium well-known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0512] Those skilled in the art should be able to realize that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented by hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. The computer-readable medium includes computer-readable storage media and communication media, where the communication media includes any medium that facilitates the transfer of a computer program from one place to another. The storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0513] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them fall within the protection scope of the present application.

Claims

1. A method for encoding metadata of multimedia, characterized in that, The method includes: Obtaining metadata of multimedia; wherein, the metadata of the multimedia includes first data, and the first data is floating-point data; Writing second data into a bitstream; wherein, the second data is a representation form of the first data in a floating-point data format.

2. The method according to claim 1, wherein The method further includes: Generating the second data based on the first data.

3. The method according to claim 2, wherein The method further includes: Obtaining the floating-point data format; The generating the second data based on the first data includes: Processing the first data according to the floating-point data format to obtain the second data.

4. The method according to claim 3, characterized in that, The method further includes: Obtaining the bit width of the second data; The processing the first data according to the floating-point data format to obtain the second data includes: Processing the first data according to the floating-point data format and the bit width to obtain the second data.

5. The method according to claim 4, wherein The method further includes: Writing the bit width into the bitstream.

6. The method according to any one of claims 1 to 5, characterized in that The method further includes: Writing the floating-point data format into the bitstream.

7. The method according to any one of claims 1 to 6, characterized in that The second data is a binary representation form or a hexadecimal representation form of the first data in a floating-point data format.

8. The method according to any one of claims 1 to 7, characterized in that, The metadata of the multimedia is metadata of a high dynamic range (HDR) image.

9. The method according to any one of claims 1 to 7, characterized in that, The metadata of the multimedia is metadata of enhanced data, and the enhanced data is generated based on a standard dynamic range (SDR) image and a high dynamic range (HDR) image.

10. The method according to any one of claims 1 to 9, characterized in that, The floating-point data format includes at least one of the following: IEEE 754, FP8 (E4M3), FP8 (E5M2), Bfloat16, TensorFloat-32.

11. A bitstream, characterized in that, The bitstream is generated according to the method of any one of claims 1 to 10 above.

12. A bitstream, characterized in that, The bitstream includes second data, and the second data is a representation form of first data in a floating-point data format, and the first data is floating-point data in the metadata of the multimedia.

13. The bitstream according to claim 12, wherein, The bitstream further includes the floating-point data format.

14. The bitstream according to claim 12 or 13, characterized in that, The second data is obtained by processing the first data according to the floating-point data format and the bit width of the second data, and the bitstream further includes the bit width.

15. A method for decoding metadata of multimedia, characterized in that, The method includes: Receiving a bitstream; Parsing second data from the bitstream, where the second data is a representation form of first data in a floating-point data format, and the first data is floating-point data in the metadata of the multimedia; Generating the first data based on the second data.

16. The method according to claim 15, wherein The method further includes: Obtaining the floating-point data format; The generating the first data based on the second data includes: Processing the second data according to the floating-point data format to obtain the first data.

17. The method according to claim 16, wherein The obtaining the floating-point data format includes: Parsing the floating-point data format from the bitstream.

18. The method according to claim 16 or 17, characterized in that The method further includes: Obtaining the bit width of the second data; The processing the second data according to the floating-point data format to obtain the first data includes: Processing the second data according to the floating-point data format and the bit width to obtain the first data.

19. The method according to claim 18, wherein Obtaining the bit width of the second data includes: Parsing the bit width from the bitstream.

20. The method according to any one of claims 15 to 19, characterized in that The second data is a binary representation or a hexadecimal representation of the first data in floating-point data format.

21. The method according to any one of claims 15 to 20, characterized in that The metadata of the multimedia is metadata of a high dynamic range (HDR) image.

22. The method according to any one of claims 15 to 21, characterized in that, The metadata of the multimedia is enhanced data metadata, and the enhanced data is determined based on a standard dynamic range (SDR) image and a high dynamic range (HDR) image.

23. The method according to any one of claims 15 to 22, characterized in that, The floating-point data format includes at least one of the following: IEEE 754, FP8 (E4M3), FP8 (E5M2), Bfloat16, TensorFloat-32.

24. An encoder, characterized in that, Including: A memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the encoder is caused to execute the method according to any one of claims 1 to 10.

25. A decoder, characterized in that, Including: A memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoder is caused to execute the method according to any one of claims 15 to 23.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor is caused to execute the method according to any one of claims 1 to 10. Or, execute the method according to any one of claims 15 to 23.

27. A computer program product, characterized in that, The computer program product includes computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 10 are caused to be executed, or, the steps of the method according to any one of claims 15 to 23 are caused to be executed.

28. A computer-readable storage medium, characterized in that The computer-readable storage medium stores the bitstream according to any one of claims 11 to 14 above.

Images