Image processing method and device

Abstract

The embodiment of the invention provides an image processing method and device. The method comprises the following steps: performing image processing on a first double-layer code stream, and outputting a second double-layer code stream. Wherein the first double-layer code stream comprises first coded data used for representing the first basic image and second coded data used for representing the first enhanced image. The second double-layer code stream comprises third coded data used for representing the second basic image and fourth coded data used for representing the second enhanced image. The first basic image and the first enhanced image are used for synthesizing a first synthesized image, the second basic image and the second enhanced image are used for synthesizing a second synthesized image, and difference associated with image processing operation exists between the first synthesized image and the second synthesized image. According to the invention, image processing of the double-layer code stream can be realized, so that the output double-layer code stream meets different scene requirements.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 15, 2023, with application number 202311037360.4 and application name “Image Processing Method and Device”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The embodiments of the present application relate to the field of image processing, and in particular, to an image processing method and device. Background Art

[0003] At present, the image processing process generally includes front-end processing (such as image generation and image encoding), intermediate processing (such as transcoding and re-encoding), and back-end processing (such as decoding and display). However, the current image processing process is usually for a single-layer bitstream, for example, only supporting HDR (High Dynamic Range) images or standard dynamic range (SDR) images. How to process the dual-layer bitstream has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides an image processing method and device. In the method, a transcoding end can perform image processing on a double-layer code stream to output a double-layer code stream that meets scene requirements.

[0005] In the first aspect, the present application provides an image processing method. The method includes: a transcoding end obtains a first double-layer code stream, the first double-layer code stream includes first coded data and second coded data, the first coded data is used to represent a first basic image, the second coded data is used to represent a first enhanced image, and the first basic image and the first enhanced image are used to synthesize a first synthetic image. The transcoding end performs an image processing operation based on the first double-layer code stream to obtain third coded data and fourth coded data, the third coded data is used to represent a second basic image, the fourth coded data is used to represent a second enhanced image, the second basic image and the second enhanced image are used to synthesize a second synthetic image, and there is a difference associated with the image processing operation between the second synthetic image and the first synthetic image. The transcoding end outputs a second double-layer code stream, and the second double-layer code stream includes the third coded data and the fourth coded data. In this way, the image processing method provided in the embodiment of the present application can be applied to a double-layer code stream scenario to output a double-layer code stream that meets the requirements of different scenarios based on the received double-layer code stream.

[0006] Exemplarily, the transcoding end may also be a transcoding device, an image processing device, etc., which is not limited in this application.

[0007] Exemplarily, the first coded data can be understood as being obtained after encoding the first basic image, the second coded data is obtained after encoding the second enhanced image, the third coded data is obtained after encoding the second basic image, and the fourth coded data is obtained after encoding the second enhanced image. Optionally, the encoding operation of the first basic image and the second enhanced image can be performed by the transmitting end.

[0008] Exemplarily, the first double-layer code stream obtained by the transcoding end may be obtained from the transmitting end. Optionally, the transmitting end and the transcoding end may belong to the same device or may be independent devices. For example, the transmitting end may be a terminal device, and the transcoding end may be a server, which is not limited in this application.

[0009] Exemplarily, the first double-layer code stream and the second double-layer code stream may further include metadata.

[0010] Exemplarily, the first double-layer code stream and the second double-layer code stream may follow the same or different coding formats.

[0011] In a possible implementation, the method also includes: the transcoding end decodes the first encoded data to obtain a first basic image, and decodes the second encoded data to obtain a first enhanced image. The transcoding end merges the first basic image and the first enhanced image to obtain a first composite image. In this way, the transcoding end can obtain the first composite image on this side, which can also be understood as the transcoding end performing the same decoding and synthesis operations as the receiving end, and can also be understood as performing the opposite operation to the sending end. That is, the sending end obtains the first enhanced image based on the first basic image and the first composite image. The receiving end obtains the first composite image based on the first basic image and the first enhanced image. In the present application, the transcoding end obtains the first composite image based on the first basic image and the first enhanced image, and processes the first composite image in the subsequent image processing process, so that after the receiving end receives the second double-layer code stream and obtains the first composite image, what can be displayed is the first composite image processed by the transcoding end.

[0012] In a possible implementation, an image processing operation is performed based on the first double-layer code stream to obtain third encoded data and fourth encoded data, including: performing an image processing operation on the first composite image to obtain a second composite image. Based on the second composite image, a second basic image is obtained. The third encoded data and the fourth encoded data are obtained based on the second composite image and the second basic image. In this way, the transcoding end of the present application can perform corresponding image processing operations based on different scene requirements to meet the requirements of different scenes for image quality or image content. Moreover, the transcoding end can make the second composite image, the second basic image and the second enhanced image have the same image features as the image processing operation (for example, they are all enlarged or watermarked) to maintain image consistency by performing an image processing operation on the first composite image and then obtaining the second basic image based on the processed second composite image.

[0013] In a possible implementation, an image processing operation is performed based on the first double-layer code stream to obtain third encoded data and fourth encoded data, including: performing an image processing operation on the first composite image to obtain a second composite image. Performing an image processing operation on the first basic image to obtain a second basic image. The third encoded data and the fourth encoded data are obtained based on the second composite image and the second basic image. In this way, the transcoding end of the present application can perform corresponding image processing operations based on different scene requirements to meet the requirements of different scenes for image quality or image content. In addition, the transcoding end performs image processing operations on the first basic image and the first composite image respectively, so that the second composite image, the second basic image and the second enhanced image have the same image features as the image processing operation (for example, they are all enlarged or watermarked), so as to maintain the consistency of the image.

[0014] In a possible implementation, obtaining the third coded data and the fourth coded data based on the second composite image and the second basic image includes: obtaining the second enhanced image based on the second composite image and the second basic image. Encoding the second basic image to obtain the third coded data. Encoding the second enhanced image to obtain the fourth coded data. In this way, the second enhanced image is obtained based on the second composite image and the second basic image after image processing, so that the enhanced image has the same (or related) image features as the second basic image and the second composite image, and the image features are image features associated with the image processing operation.

[0015] In a possible implementation, obtaining third coded data and fourth coded data based on the second composite image and the second basic image includes: encoding the second basic image to obtain the third coded data. Obtaining fourth coded data based on the third coded data and the second composite image.

[0016] In a possible implementation, performing an image processing operation based on the first code stream to obtain third coded data and fourth coded data includes: performing an image processing operation on the first composite image to obtain a second composite image. The second basic image is the first basic image. The third coded data and the fourth coded data are obtained based on the second composite image and the second basic image.

[0017] Exemplarily, the transcoding end may input the first basic image into the image processing module, the image processing module does not process the first basic image, and the second basic image output by the image processing module is the first basic image. Of course, in some examples, the first basic image may not be input into the image processing module, but directly input into the back end of the image processing module as the second basic image to continue to perform subsequent operations.

[0018] In a possible implementation, performing an image processing operation based on the first code stream to obtain third coded data and fourth coded data includes: the second composite image is the first composite image. Performing an image processing operation on the first basic image to obtain a second basic image. The third coded data and the fourth coded data are obtained based on the second composite image and the second basic image. In this way, when performing the image processing operation, the transcoding end can perform the image processing operation on different objects to meet different image quality requirements for different scenes.

[0019] Exemplarily, the transcoding end may input the first composite image into the image processing module, the image processing module does not process the first composite image, and the second composite image output by the image processing module is the first composite image. Of course, in some examples, the first composite image may not be input into the image processing module, but may be directly input into the back end of the image processing module as the second composite image to continue to perform subsequent operations.

[0020] In a possible implementation, performing an image processing operation based on the first code stream to obtain the third coded data and the fourth coded data includes: performing an image processing operation on the first basic image to obtain the second basic image; obtaining a second composite image based on the second basic image; and obtaining the third coded data and the fourth coded data based on the second composite image and the second basic image. In this way, when performing image processing operations, the transcoding end can perform image processing operations on different objects to meet different image quality requirements for different scenes.

[0021] In a possible implementation, performing an image processing operation based on the first code stream to obtain third coded data and fourth coded data includes: performing an image processing operation on a first basic image to obtain a second basic image. Performing an image processing operation on a first enhanced image to obtain a second enhanced image. The third coded data and the fourth coded data are obtained based on the second enhanced image and the second basic image. In this way, when performing the image processing operation, the transcoding end can perform the image processing operation on different objects to meet different image quality requirements for different scenes.

[0022] In a possible implementation, obtaining third coded data and fourth coded data based on the second enhanced image and the second basic image includes: encoding the second basic image to obtain the third coded data. Encoding the second enhanced image to obtain the fourth coded data.

[0023] In a possible implementation, the operation mode of the image processing operation is obtained, and the operation mode includes at least one of the following: enlargement, reduction, sharpening, passivation, adding watermarks, or removing watermarks. In this way, the transcoding end of the present application can provide different image processing modes to meet the needs of different scenarios.

[0024] In a possible implementation, the operation mode is preset, or the operation mode of the image processing operation is obtained, including: determining the operation mode based on the received user operation or user demand; or obtaining the operation mode from the first double-layer code stream. In this way, the transcoding end can process the image based on the demand of the sending end, the demand of the user and / or the demand preset by the transcoding end, so as to output a double-layer code stream that meets the requirements of different scenarios.

[0025] In a possible implementation, the difference includes at least one of the following: the resolution of the second composite image is different from that of the first composite image. Or, the clarity of the second composite image is different from that of the first composite image. Or, the image content of the second composite image has more or less watermarks than that of the first composite image.

[0026] In a second aspect, the present application provides an image processing device, comprising: an acquisition module, used to acquire a first double-layer code stream, the first double-layer code stream includes first coded data and second coded data, the first coded data is used to represent a first basic image, the second coded data is used to represent a first enhanced image, the first basic image and the first enhanced image are used to synthesize a first synthetic image. An image processing module, used to perform an image processing operation based on the first double-layer code stream to obtain third coded data and fourth coded data, the third coded data is used to represent a second basic image, the fourth coded data is used to represent a second enhanced image, the second basic image and the second enhanced image are used to synthesize a second synthetic image, and there is a difference associated with the image processing operation between the second synthetic image and the first synthetic image. The image processing module is also used to output a second double-layer code stream, the second double-layer code stream includes the third coded data and the fourth coded data.

[0027] In a possible implementation, the device further includes: a decoding module, configured to decode the first coded data to obtain a first basic image, and decode the second coded data to obtain a first enhanced image; and a synthesis module, configured to merge the first basic image and the first enhanced image to obtain a first synthesized image.

[0028] In a possible implementation, the image processing module is specifically used to perform an image processing operation on the first composite image to obtain a second composite image, and based on the second composite image, obtain a second basic image. The device also includes a coding module, which is used to obtain third coded data and fourth coded data based on the second composite image and the second basic image.

[0029] In one possible implementation, the image processing module is specifically used to perform image processing operations on the first composite image to obtain a second composite image; perform image processing operations on the first basic image to obtain a second basic image; the device also includes an encoding module for obtaining third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0030] In a possible implementation, the image processing module is specifically used to obtain a second enhanced image based on the second composite image and the second basic image; the device also includes an encoding module, which is used to encode the second basic image to obtain third encoded data. Encode the second enhanced image to obtain fourth encoded data.

[0031] In a possible implementation manner, the encoding module is specifically configured to encode the second basic image to obtain third encoded data; and obtain fourth encoded data based on the third encoded data and the second composite image.

[0032] In a possible implementation, the image processing module is further configured to perform an image processing operation on the first composite image to obtain a second composite image. The second basic image is the first basic image. The encoding module is configured to obtain third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0033] In a possible implementation, the second composite image is the first composite image, the image processing module is used to perform image processing operations on the first basic image to obtain the second basic image, and the encoding module is used to obtain third encoding data and fourth encoding data based on the second composite image and the second basic image.

[0034] In one possible implementation, the image processing module is used to perform image processing operations on the first basic image to obtain a second basic image; based on the second basic image, a second composite image is obtained; and the encoding module is used to obtain third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0035] In a possible implementation, the image processing module is used to perform an image processing operation on the first basic image to obtain a second basic image. The image processing operation is performed on the first enhanced image to obtain a second enhanced image. The encoding module is used to obtain third encoded data and fourth encoded data based on the second enhanced image and the second basic image.

[0036] In a possible implementation, the encoding module is used to encode the second basic image to obtain third encoded data and to encode the second enhanced image to obtain fourth encoded data.

[0037] In a possible implementation, an operation mode of an image processing operation is obtained, where the operation mode includes at least one of the following: enlarging, reducing, sharpening, blunting, adding a watermark, or removing a watermark.

[0038] In a possible implementation, the operation mode is preset, or the operation mode of the image processing operation is obtained, including: determining the operation mode based on a received user operation or user demand; or obtaining the operation mode from the first dual-layer code stream.

[0039] In a possible implementation, the difference includes at least one of the following: the resolution of the second composite image is different from that of the first composite image. Or, the clarity of the second composite image is different from that of the first composite image. Or, the image content of the second composite image has more or less watermarks than that of the first composite image.

[0040] In a third aspect, the present application provides an image processing device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the following steps: performing an image processing operation based on a first double-layer code stream to obtain third coded data and fourth coded data, the third coded data is used to represent a second basic image, the fourth coded data is used to represent a second enhanced image, the second basic image and the second enhanced image are used to synthesize a second composite image, and there is a difference associated with the image processing operation between the second composite image and the first composite image. Output a second double-layer code stream, the second double-layer code stream includes the third coded data and the fourth coded data.

[0041] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: decoding the first encoded data to obtain a first basic image, and decoding the second encoded data to obtain a first enhanced image; merging the first basic image and the first enhanced image to obtain a first composite image.

[0042] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: perform image processing operations on the first composite image to obtain a second composite image; based on the second composite image, obtain a second basic image; based on the second composite image and the second basic image, obtain third encoded data and fourth encoded data.

[0043] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: perform image processing operations on the first composite image to obtain a second composite image; perform image processing operations on the first basic image to obtain a second basic image; and obtain third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0044] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: obtain a second enhanced image based on the second composite image and the second basic image; encode the second basic image to obtain third encoded data; encode the second enhanced image to obtain fourth encoded data.

[0045] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: encode the second basic image to obtain third encoded data; and obtain fourth encoded data based on the third encoded data and the second composite image.

[0046] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: performing image processing operations on the first composite image to obtain a second composite image; the second basic image is the first basic image; and obtaining third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0047] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: the second composite image is the first composite image; image processing operations are performed on the first basic image to obtain a second basic image; and third encoded data and fourth encoded data are obtained based on the second composite image and the second basic image.

[0048] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: perform image processing operations on the first basic image to obtain a second basic image; based on the second basic image, obtain a second composite image; based on the second composite image and the second basic image, obtain third encoded data and fourth encoded data.

[0049] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: perform image processing operations on the first basic image to obtain a second basic image; perform image processing operations on the first enhanced image to obtain a second enhanced image; and obtain third encoded data and fourth encoded data based on the second enhanced image and the second basic image.

[0050] In a possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: encoding the second basic image to obtain third encoded data; encoding the second enhanced image to obtain fourth encoded data.

[0051] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: obtaining an operating mode of an image processing operation, the operating mode including at least one of the following: enlarging, reducing, sharpening, blunting, adding watermarks, or removing watermarks.

[0052] In a possible implementation, the operation mode is preset, or the operation mode of the image processing operation is obtained, including: determining the operation mode based on a received user operation or user demand; or obtaining the operation mode from the first dual-layer code stream.

[0053] In a possible implementation, the difference includes at least one of the following: the resolution of the second composite image is different from that of the first composite image. Or, the clarity of the second composite image is different from that of the first composite image. Or, the image content of the second composite image has more or less watermarks than that of the first composite image.

[0054] In a fourth aspect, the present application provides a computer-readable storage medium, comprising a computer program, wherein when the computer program is executed on a computer, the computer executes any one of the methods described in the first aspect.

[0055] In a fifth aspect, the present application provides a computer program, which, when executed by a computer, is used to perform any of the methods described in the first aspect.

[0056] In a sixth aspect, the present application also provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the operations and / or processing performed by the electronic device in any of the above method embodiments.

[0057] In a seventh aspect, an embodiment of the present application provides a code stream structure, which includes encoded data of a first basic image, encoded data of a first enhanced image, and metadata.

[0058] In an eighth aspect, an embodiment of the present application provides a code stream structure, which includes encoded data of a first basic image, encoded data of a first enhanced image, and metadata, wherein the metadata includes tone mapping information. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic block diagram of a coding solution system used in an embodiment of the present application;

[0060] Figure 2A A block diagram of a content providing system is shown as an example;

[0061] Figure 2B is a schematic diagram of the structure of a terminal device shown as an example;

[0062] Figure 3A The following is a schematic diagram of a streaming media system workflow shown as an example;

[0063] Figure 3B The following is a schematic diagram of an exemplary streaming media system architecture;

[0064] Figure 4A is an architectural diagram of an image processing system shown as an example;

[0065] Figure 4Bis an architectural diagram of an image processing system shown as an example;

[0066] Figure 5 A schematic diagram showing a dynamic mapping of a high dynamic range in the real world to a low dynamic range of a display device;

[0067] Figure 6 is a schematic diagram of an end-to-end system of a dual-layer code stream shown as an example;

[0068] Fig. 7A A system block diagram of image processing is shown as an example;

[0069] Figure 7B is a system block diagram of another image processing system shown as an example;

[0070] Figure 8 is a flowchart of an exemplary image processing method;

[0071] Fig. 9 is a schematic diagram of an exemplary tone mapping process;

[0072] Fig.10 FIG. 1 is a schematic diagram of the structure of an image processing device shown as an example. DETAILED DESCRIPTION

[0073] The term "at least one" in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, wherein any one of a, b, c, ab, ac, bc, or abc can include a single a, a single b, a single c, and can also include multiple a, multiple b, and multiple c.

[0074] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria, and do not indicate the difference in the content, priority or importance of the two criteria.

[0075] In addition, the terms "including" and "having" in the embodiments, claims and drawings of the present application are not exclusive. For example, a process, method, system, product or device including a series of steps or modules / units is not limited to the listed steps or modules, and may also include steps or modules / units that are not listed.

[0076] First, a brief description of the background technology that may be involved in this application is given:

[0077] Figure 1 A schematic block diagram of a coding and decoding system used in an embodiment of the present application. The video encoder 20 (or simply encoder 20) and the video decoder 30 (or simply decoder 30) in the video coding and decoding system 10 represent devices that can be used to perform various techniques according to various examples described in this application.

[0078] like Figure 1 As shown, the encoding and decoding system 10 includes a source device 12, and the source device 12 is used to provide encoded data 21 such as encoded images to a destination device 14 for decoding the encoded data.

[0079] The source device 12 includes an encoder 20 , and optionally may further include an image source 16 , an image preprocessor 18 (or a preprocessing unit), and a communication interface or a communication unit 22 .

[0080] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0081] In order to distinguish the processing performed by the pre-processor 18 or the pre-processing unit 18 , the image or the image data 17 may also be referred to as a raw image or raw image data 17 .

[0082] The preprocessor 18 is used to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment, or denoising. It is understood that the preprocessing unit 18 may be an optional component.

[0083] The video encoder 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 .

[0084] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device (such as a transcoding end) through the communication channel 13 for storage or direct reconstruction.

[0085] Destination device 14 includes a decoder 30 (eg, video decoder 30 ) and, in addition or alternatively, may include a communication interface or communication unit 28 , a post-processor 32 (or post-processing unit 32 ), and a display device 34 .

[0086] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a coded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0087] The communication interface 22 and the communication interface 28 can be used to send or receive the encoded image data 21 or the encoded data through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0088] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0089] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21 .

[0090] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1 A unidirectional communication interface, or a bidirectional communication interface, is indicated by an arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0091] The decoder 30 is configured to receive the encoded image data 21 and provide decoded image data 31 or a decoded image 31 .

[0092] The post-processor 32 in the destination device 14 is used to post-process the decoded image data 31 (also called reconstructed image data) such as the decoded image 31 to obtain the post-processed image data 33 such as the post-processed image 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., conversion from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing for generating the decoded image data 31 for display by the display device 34 or the like.

[0093] The display device 34 in the destination device 14 is used to receive the post-processed image data 33 to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0094] although Figure 1 The source device 12 and the destination device 14 are shown as independent devices, but the device embodiment may also include both the source device 12 and the destination device 14 or the functions of both the source device 12 and the destination device 14, that is, both the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In these embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0095] According to the description, Figure 1 The presence and (exact) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, as will be apparent to the skilled person.

[0096] Figure 2A This is a block diagram of an exemplary content providing system. Figure 2AThe content providing system of the content distribution service used in the present application is described, and the content providing system 20 includes a capture device 2102, a terminal device 2106 and an (optional) display 2126. The capture device 2102 communicates with the terminal device 2106 via a communication link 2104. The communication link may include the above-mentioned communication channel 13. The communication link 2104 includes but is not limited to WiFi, Ethernet, wired, wireless (3G / 4G / 5G), USB or any combination thereof.

[0097] The capture device 2102 generates data and encodes the data. Alternatively, the capture device 2102 can distribute the data to a streaming media server (not shown in the figure), and the server encodes the data and transmits the encoded data to the terminal device 2106. The capture device 2102 includes but is not limited to a camera, a smart phone or a tablet computer, a computer or a laptop computer, a video conferencing system, a PDA, a vehicle-mounted device, or any combination thereof. For example, the capture device 2102 may include the above-mentioned source device 12. When the data includes video, the encoder 20 in the capture device 2102 can actually perform video encoding processing. When the data includes audio (i.e., sound), the encoder 20 in the capture device 2102 can actually perform audio encoding processing. In some actual scenarios, the capture device 2102 distributes the encoded video data and the encoded audio data by multiplexing the encoded video data and the encoded audio data together. In other actual scenarios, for example, in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. The capture device 2102 distributes the encoded audio data and the encoded video data to the terminal device 2106 respectively.

[0098] The terminal device 2106 in the content providing system 20 receives and regenerates the encoded data. The terminal device 2106 can be a device with data receiving and recovery capabilities, such as a smart phone or tablet computer 2108, a computer or laptop computer 2110, a network video recorder (NVR) / digital video recorder (DVR) 2112, a television 2114, a set top box (STB) 2116, a video conferencing system 2118, a video surveillance system 2120, a personal digital assistant (PDA) 2122, a vehicle-mounted device 2124 or any combination thereof, or such a device capable of decoding the above-mentioned encoded data. For example, the terminal device 2106 may include the above-mentioned destination device 14. When the encoded data includes video, the video decoder 30 in the terminal device preferentially performs video decoding. When the encoded data includes audio, the audio decoder in the terminal device preferentially performs audio decoding processing. The terminal device 2106 can be a video playback application, a streaming media playback application, a streaming media playback platform, or a live broadcast platform, etc. running on the terminal device.

[0099] For terminal devices with a display, such as a smartphone or tablet 2108, computer or laptop 2110, NVR / DVR 2112, TV 2114, PDA 2122, or vehicle-mounted device 2124, the terminal device can send the decoded data to its display. For terminal devices without a display, such as STB 2116, video conferencing system 3118, or video surveillance system 2120, connect the device to an external display 2126 to receive and display the decoded data.

[0100] When each device in the system performs encoding or decoding, the image encoding device or image decoding device shown in the above embodiments may be used.

[0101] Figure 2B for Figure 2ASchematic diagram of the example structure of the terminal device 2106 in FIG. After the terminal device 2106 receives the bit stream from the capture device 2102, the protocol processing unit 2202 analyzes the transmission protocol of the bit stream. The protocol includes but is not limited to Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG HTTP Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP) or any combination thereof.

[0102] After the protocol processing unit 2202 processes the stream, a stream file is generated. The file is output to the demultiplexing unit 2204. The demultiplexing unit 2204 can separate the multiplexed data into coded audio data and coded video data. As mentioned above, in other actual scenarios, such as in a video conferencing system, the coded audio data and the coded video data are not multiplexed. In this case, the coded data is not transmitted to the video decoder 3206 and the audio decoder 2208 through the demultiplexing unit 2204.

[0103] Through the demultiplexing process, a video elementary stream (ES), an audio ES, and optional subtitles are generated. The video decoder 2206, including the video decoder 30 described in the above embodiment, decodes the video ES to generate a video frame by the decoding method shown in the above embodiment, and sends the data to the synchronization unit 2212. The audio decoder 2208 decodes the audio ES to generate an audio frame, and sends the data to the synchronization unit 2212. Alternatively, the video frame can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212. Similarly, the audio frame can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212.

[0104] The synchronization unit 2212 synchronizes the video frames and the audio frames and provides the video / audio to the video / audio display 3214. For example, the synchronization unit 2212 synchronizes the presentation of the video and audio information. The information can be encoded in the syntax using the timestamps associated with the encoded audio and visual data representations and the timestamps associated with the data stream transmission.

[0105] If subtitles are included in the bitstream, the subtitle decoder 2210 decodes the subtitles, synchronizes them with the video frames and audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 2216 .

[0106] The present invention is not limited to the above-mentioned system, and the image encoding device or the image decoding device in the above-mentioned embodiments can be used in other systems such as automobiles.

[0107] Figure 3A This is a schematic diagram of a streaming media system workflow applicable to the embodiment of this application. Figure 3A A streaming media system to which the embodiments of the present application are applicable is described.

[0108] The streaming media system includes a content creation module to generate the required content data, such as video or audio. The streaming media system also includes a video encoding module to encode the generated content through an encoder. The streaming media system also includes a video stream transmission module to transmit the encoded video in the form of a code stream. Optionally, the video stream transcoding module can convert the format of the video stream into a code stream format of a commonly used transmission protocol for OTT (over-the-top) devices, for example, the protocol includes but is not limited to Real Time Streaming Protocol (RTSP), HyperText Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG HTTP Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real-time Messaging Protocol (RTMP) or any combination thereof. Optionally, the video stream storage module can store the video stream to store the original format of the video stream and / or the converted multiple code stream formats for easy use. Furthermore, the streaming media system also includes a video stream encapsulation module for encapsulating the video stream to generate an encapsulated video stream, which can be called a video streaming media package. Exemplarily, the video streaming media package can be generated according to the transcoded video stream or the stored video stream. Furthermore, the streaming media system also includes a content distribution network (CDN), which is used to distribute the video streaming media package to multiple OTT devices, such as mobile phones, computers, tablets, home projectors, etc.

[0109] It should be noted that video encoding, video streaming transmission, video stream transcoding, video stream storage, video streaming package generation and content distribution network can all be implemented on cloud servers.

[0110] Figure 3B The following is a schematic diagram of the streaming media system architecture. Figure 3B An exemplary streaming media system architecture of the present application is described. The streaming media system architecture includes: a client device, a content distribution network, and a cloud server.

[0111] The user on the client device sends a play or playback request to the cloud platform. Optionally, the content of the request sent may be the title of the movie or TV program being played.

[0112] The cloud platform makes a decision, replies to the client, and sends the address of the content requested by the client on the CDN to the client. Optionally, the content sent to the client can be a URL link (uniform resource locator). Specifically, the playback application service in the cloud platform checks user authorization and permission, and then considers the characteristics of each client and the current network conditions to determine which specific files are needed to process the playback request. It should be noted that the content delivery network (CDN) regularly reports the operating status, learned routes, and available content (files) to the cache control service in the cloud platform.

[0113] The client then requests the CDN to play the content based on the address, and the CDN provides the content to the client, ultimately completing the client's request.

[0114] Figure 4A This is a schematic diagram of a possible system architecture applicable to the embodiment of the present application. The system architecture of the embodiment of the present application includes: a front-end device, a transmission link, and a terminal display device. Figure 4A Describe the system architecture applicable to the embodiments of the present application.

[0115] Among them, the front-end device is used to capture or produce HDR / SDR content (for example, HDR / SDR video or image).

[0116] In a possible embodiment, the front-end device can also be used to extract corresponding metadata from the HDR content, and the metadata may include global mapping information, local mapping information, and dynamic metadata and static metadata corresponding to the HDR content. The front-end device can send the HDR content and metadata to the terminal display device through a transmission link. Specifically, the HDR content and metadata can be transmitted in the form of a data packet (i.e., an HDR code stream), or respectively transmitted in two data packets (i.e., an HDR code stream and a metadata stream), which is not specifically limited in the embodiments of the present application.

[0117] Optionally, the terminal display device can be used to receive metadata and HDR content, and extract the global mapping information, local mapping information and the terminal display device information contained in the corresponding metadata according to the HDR content, obtain a mapping curve to perform global tone mapping and local tone mapping on the HDR content, convert it into display content adapted for the HDR display device or SDR device in the terminal display device, and display it. It should be understood that in different embodiments, the terminal display device may include a display device with a display capability of a lower dynamic range or a higher dynamic range than the HDR content generated by the front-end device, and this application is not limited thereto.

[0118] Optionally, the front-end device and the terminal display device in the present application may be independent and different physical devices. For example, the front-end device may be a video acquisition device or a video production device, wherein the video acquisition device may be a video camera, a camera, an image plotter, etc. The terminal display device may be a device with a video playback function, such as virtual reality (VR) glasses, a mobile phone, a tablet, a television, a projector, etc.

[0119] Optionally, the transmission link between the front-end device and the terminal display device can be a wireless connection or a wired connection, wherein the wireless connection can adopt technologies such as long term evolution (LTE), fifth generation (5G) mobile communication and future mobile communication. Wireless connection can also include wireless fidelity (WiFi), Bluetooth, near field communication (NFC) and other technologies. Wired connection can include Ethernet connection, local area network connection, etc. There is no specific limitation on this.

[0120] The present application may also integrate the functions of the front-end device and the functions of the terminal display device on the same physical device, for example, a terminal device such as a mobile phone or a tablet with a video shooting function. The present application may also integrate some functions of the front-end device and some functions of the terminal display device on the same physical device. There is no specific limitation on this.

[0121] Figure 4A The specific implementation of the end-to-end system in Figure 1 , I will not go into details here.

[0122] Figure 4B A schematic diagram of the structure of an image processing system provided in an embodiment of the present application is shown below. Figure 4B Describe the end-to-end image processing system provided by the embodiment of the present application, which can be applied to Figure 4A In the system architecture shown in Figure 4BIn the example, HDR / SDR content takes HDR video as an example. The image processing system includes: an HDR preprocessing module, an HDR video encoding module, an HDR video decoding module, and a tone mapping module. Optionally, the system may also include a transcoding module.

[0123] Among them, the HDR preprocessing module and the HDR video encoding module can be located in Figure 4A The front-end device shown in the figure, the HDR video decoding module and the tone mapping module can be located in Figure 4A The terminal display device shown.

[0124] HDR preprocessing module: used to extract dynamic metadata (e.g., maximum, minimum, average, and range of brightness) from HDR video, and determine mapping curve parameters based on dynamic metadata and the display capability of the target display device, write the mapping curve parameters into the dynamic metadata to obtain HDR metadata, and transmit it. HDR video can be acquired or processed by a colorist; the display capability of the target display device is the brightness range that the target display device can display.

[0125] HDR video encoding module: used to encode HDR video and HDR metadata according to the video compression standard (for example, AVS or HEVC standard) (for example, embedding HDR metadata into the user-defined part of the bitstream) and output the corresponding bitstream (AVS or HEVC bitstream).

[0126] Transcoding module: used to receive bitstreams. Transcode the bitstream. Transcoding can optionally be to convert a compressed and encoded bitstream into another bitstream to adapt to different network bandwidths, different terminal processing capabilities, and / or different user needs. Transcoding can be understood as a process of first decoding the bitstream and then re-encoding it. Optionally, the bitstreams before and after transcoding may follow the same video coding standard or different video coding standards, which is not limited in this application and will not be repeated below.

[0127] HDR video decoding module: used to decode the generated bitstream (AVS bitstream or HEVC bitstream) according to the standard corresponding to the bitstream format, and output the decoded HDR video and HDR metadata.

[0128] Tone mapping module: used to generate a mapping curve according to the parameters of the mapping curve in the decoded HDR metadata, and perform tone mapping on the decoded HDR video (i.e. HDR adaptation processing or SDR adaptation processing), and send the HDR adapted video after tone mapping to the HDR display terminal for display, or send the SDR adapted video to the SDR display terminal for display.

[0129] Exemplarily, the HDR pre-processing module may exist in a video acquisition device or a video production device.

[0130] Exemplarily, the HDR video encoding module may exist in a video acquisition device or a video production device.

[0131] Exemplarily, the HDR video decoding module may exist in a set-top box, a television display device, a mobile terminal display device, and a video conversion device for online live broadcasting, online video applications, and the like.

[0132] Exemplarily, the tone mapping module may exist in a set-top box, a television display device, a mobile terminal display device, and a video conversion device for a live webcast, a web video application, etc. More specifically, the tone mapping module may exist in the form of a chip or a software program in a set-top box, a television display, a mobile terminal display, and may exist in the form of a software program in a video conversion device for a live webcast, a web video application, etc.

[0133] In a possible embodiment, when the tone mapping module and the HDR video decoding module are both present in a set-top box, the set-top box can complete the functions of receiving, decoding, and tone mapping the video code stream, and the set-top box sends the decoded video data to a display device through a high definition multimedia interface (HDMI) for display, so that the user can enjoy the video content.

[0134] The following is a description of the technical terms that may be involved in this application:

[0135] 1) High Dynamic Range (HDR)

[0136] Dynamic Range is the ratio of the maximum brightness to the minimum brightness in a video or image signal. Dynamic Range is used in many fields to indicate the ratio of the maximum value to the minimum value of a variable. In digital images, the dynamic range indicates the ratio between the maximum grayscale value and the minimum grayscale value within the displayable range of the image. The dynamic range in nature is quite large. The brightness of a night scene under the starry sky is about 0.001cd / m2, and the brightness of the sun itself is as high as 1,000,000,000cd / m2. Such a dynamic range reaches the order of 1,000,000,000 / 0.001=1013. However, in real scenes in nature, the brightness of the sun and the brightness of the stars will not be obtained at the same time. For natural scenes in the real world, the dynamic range is in the range of 10-3 to 106. In most color digital images, each channel of R, G, and B is stored in an 8-bit byte. That is to say, the range of each channel is 0 to 255 grayscale levels. Here, 0 to 255 is the dynamic range of the image. Since the dynamic range of the same scene in the real world is in the range of 10-3 to 106, we call it High Dynamic Range (HDR). The dynamic range of ordinary pictures is Low Dynamic Range (LDR). The imaging process of a digital camera is actually the mapping of the high dynamic range of the real world to the low dynamic range of the photo. The mapping of the high dynamic range in the real world to the low dynamic range of the photo is often a nonlinear process.

[0137] Figure 5 A schematic diagram showing a dynamic mapping of a high dynamic range in the real world to a low dynamic range of a display device. Figure 5 The real-world dynamic range is around 80 to 2000, and the dynamic range mapped to the display device is around 1 to 200.

[0138] 2) Standard dynamic range (SDR) images and their photoelectric conversion functions

[0139] Standard dynamic range (SDR) images, also known as low dynamic range, are images with a dynamic range between 1nit and 100nit. Images with a dynamic range between 1nit and 100nit generally conform to BT.709 or sRGB color gamut and use the gamma curve as the photoelectric conversion curve.

[0140] Standard dynamic range images correspond to high dynamic range images. The 8-bit images in the traditional jpeg format can be considered as standard dynamic range images. Before the emergence of cameras that can shoot HDR images, traditional cameras can only record the captured light information within a certain range by controlling the exposure value. Since the maximum brightness information of the display device cannot reach the brightness information of the real world, and we browse images through the display device, a photoelectric transfer function is required. The early display device was a cathode ray tube (CRT) display, and its photoelectric transfer function was the Gamma function. This photoelectric transfer function based on the "Gamma" function is defined in the ITU-R Recommendation BT.1886 standard.

[0141] 3) HDR photoelectric conversion function

[0142] With the upgrade of display devices, the illumination range of display devices continues to increase. The existing consumer-level HDR display is 600cd / m2, and the illumination information of high-end HDR display can reach 2000cd / m2, which far exceeds the illumination information of SDR display devices. The photoelectric conversion function in the ITU-R Recommendation BT.1886 standard cannot well express the display performance of HDR display devices. Therefore, an improved photoelectric transfer function is needed to adapt to the upgrade of display devices. At present, there are three common photoelectric conversion functions: Perception quantization (PQ), Hybrid Log-Gamma (HLG) and Scene Luminance Fidelity (SLF) photoelectric conversion functions. The above three curves can refer to the existing technology, and this application will not repeat them.

[0143] 4) HDR end-to-end process

[0144] The dynamic range mapping method is mainly to adapt the front-end HDR signal and the back-end HDR terminal display device through dynamic range mapping. For example, the front end collects a 4000nit light signal (nit is the unit of light intensity), while the back-end HDR terminal display device (TV, IPAD) has an HDR display capability of only 500nit. Therefore, how to map the 4000nit signal to the 500nit device is a high-to-low tone-mapping (tone mapping, TM) process. Another is that the front end collects a 100nit SDR signal, while the display end is a 2000nit TV signal. If the 100nit signal is better displayed on a 2000nit device, it is another low-to-high tone-mapping process.

[0145] Dynamic range mapping can be divided into static and dynamic. The static mapping method is to use a single piece of data to perform the overall tone-mapping process based on the same video content or the same hard disk content, that is, the processing curve is usually the same. The advantage of this method is that it carries less information and the processing flow is relatively simple; the disadvantage of this method is that each scene uses the same curve for tone-mapping, which will cause information loss in some scenes. For example, if the curve focuses on protecting the bright area, some details will be lost in some extremely dark scenes, or simply invisible, which will affect the experience. The dynamic mapping method is to make dynamic adjustments based on the content of a specific area, each scene, or each frame. The advantage of this method is that different curves can be processed according to specific areas, each scene, or each frame, so that the processing results will be better. The disadvantage is that each frame or each scene needs to carry relevant scene information, and the amount of information carried is large.

[0146] 5) A dual-layer code stream, including basic image coding data (for example, the first coding data and the third coding data in the embodiment of the present application) and enhanced image coding data (for example, the second coding data and the fourth coding data in the embodiment of the present application). Among them, the enhanced image coding data is used to characterize the enhanced image (which may also be referred to as the enhanced layer image, etc., which is not limited in the present application), and the basic image coding data is used to characterize the basic image (for example, which may also be referred to as the basic layer image, the basic image, the basic layer image, etc., which is not limited in the present application). In the embodiment of the present application, the dual-layer code stream may include one or more enhanced images, and one or more enhanced images. In the process of processing the image (including enhanced image data or basic image data) in the following embodiments, unless otherwise specified, it means that the same processing is performed on each image data. In the embodiment of the present application, the dual-layer code stream may be an image (or picture) code stream or a video code stream. That is, the enhanced image and the basic image in the dual-layer code stream may be a picture or a video, that is, the image processing method in the embodiment of the present application may be applied to the processing of the image code stream or the processing of the video code stream. Optionally, in some examples, the double-layer code stream can also be understood as encapsulating the enhanced image code stream and the basic image code stream into one code stream, that is, after the receiving end or the transcoding end receives the double-layer code stream and processes the double-layer code stream (which can be called decapsulation processing), the basic image code stream and the enhanced image code stream can be obtained. In other words, the various processing of the enhanced image described in the embodiments of the present application (including decoding, image processing and encoding, etc.) can be understood as processing the enhanced image stream (i.e., one or more enhanced images). The various processing of the basic image (including decoding, image processing and encoding, etc.) can be understood as processing the basic image stream (i.e., one or more basic images), which is not limited in this application and will not be repeated below.

[0147] 6) Single-layer code stream, which can be an enhanced image code stream or a basic image code stream. Among them, the enhanced image code stream includes one or more enhanced images. The basic image code stream includes one or more basic images. Optionally, the HDR video (or HDR image) in the existing technology can be encoded and transmitted using H.265, etc., and the single-layer video (i.e., single-layer code stream) is transcoded during the transmission process. After decoding, the HDR video obtains frame-by-frame HDR pixel values ​​and HDR format identifiers and information. HDR video usually needs to be sent to the image processing module to complete image up and down sampling, enhancement and other operations. The processed HDR pixel values ​​and HDR format identifiers and information are then sent to the encoder for re-encoding.

[0148] 7) Base image: describes an independent image data structure, including pixels and image-related metadata.

[0149] 8) Enhancement image: describes the enhanced image data structure, including pixel and image-related metadata.

[0150] 9) Synthetic image, which may also be referred to as a derived alternate image, a selectable image, a replacement image, a substitute image, etc. in the Ultra High Definition World Association (UWA) standard, and is not limited in this application. Synthetic image is used to describe the image data structure obtained from the format specified in the embodiment of this application for display or subsequent processing, including metadata related to pixels and images.

[0151] 10) Metadata: data that describes image attributes and characteristics, as well as key information required during image processing.

[0152] Figure 6 FIG. 1 is a schematic diagram showing an exemplary end-to-end system of a dual-layer code stream. Figure 6 As shown, a two-layer code stream end-to-end processing system (also referred to as a two-layer distribution format end-to-end system, which is not limited in this application) includes but is not limited to: a sending end (also referred to as an encoding end, a generating end, etc., which is not limited in this application), a transcoding end and a receiving end (also referred to as a decoding end or a display end, which is not limited in this application).

[0153] At the generation end, the enhanced image generation module obtains DRE-AIG (dynamic range extending associated image group, dynamic range extended associated image) data (including basic image, enhanced image and metadata) from the basic image data and the synthesized image data. Next, the encoding module generates a basic image code stream, an enhanced image code stream and a metadata code stream. Then, a double-layer code stream (which can be understood as the first double-layer code stream described in the embodiment of the present application) is generated for the encapsulated image and metadata code stream, which can also be called an HDR static image double-layer distribution format file.

[0154] In the embodiment of the present application, the generating end transmits the first double-layer code stream to the transcoding end (also referred to as the distributing end).

[0155] At the transcoding end, the first double-layer code stream is received. The first double-layer code stream is transcoded and outputted (which can be understood as the second double-layer code stream described in the embodiment of the present application). The specific operation process will be described in detail in the following embodiment. The first double-layer code stream and the second double-layer code stream may follow the same or different encoding formats, which is not limited in the present application.

[0156] At the receiving end, after receiving the second double-layer code stream, the code stream is decapsulated to extract the basic image code stream, the enhanced image code stream and the metadata code stream. Then, the basic image code stream and the enhanced image code stream are decoded respectively to obtain DRE-AIG data, enhanced image data and metadata. Then, the synthesized image is obtained, and display adaptation processing is performed before display (the display end can refer to the above, which will not be repeated here).

[0157] Optionally, in some examples, the transcoding end processes the code stream and saves or transmits it to the decoding end. Optionally, the encoding end and the transcoding end can be set in different devices or the same device (for example, both are set in the sending end), which is not limited in this application.

[0158] Fig. 7A : is a system block diagram of an exemplary image processing. An embodiment of the present application provides an image processing method suitable for a transcoding end. As described above, transcoding may optionally be converting a compressed and encoded code stream into another code stream to adapt to different network bandwidths, different terminal processing capabilities, and / or different user needs. In the process of image processing of image data by the transcoding end in the embodiment of the present application, the image processing may be performed on the double-layer code stream according to the requirements of different scenarios, so as to meet the requirements of the transcoded double-layer code stream in different scenarios. For example, in some scenarios, the third-party platform to which the transcoding end belongs may perform processing such as downsampling on the image to reduce the resolution of the image, taking into account the storage cost, so as to reduce the storage space occupied by the image data and reduce the bandwidth occupied by the transmission. For another example, in some scenarios, if the platform has high requirements on the quality of the image data, the resolution of the image data may be increased during the transcoding process to improve the quality of the transcoded image data.

[0159] In the embodiment of the present application, the image processing method corresponds to the scene requirement, which may be set by the user, or by the platform (i.e., pre-set by the transcoding end, or by the operator of the platform), or by the sending end, which is not limited in the present application. For a specific description, please refer to the following.

[0160] Please refer to Fig. 7A The transcoding end (which may also be referred to as a transcoding device, a transcoding equipment, a transcoding module, a transcoding server, etc., which is not limited in this application) may optionally include but is not limited to: a decapsulation module, a decoding module, an image processing module, an encoding module and an encapsulation module, etc.

[0161] At the transcoding end, a first code stream (also referred to as a first double-layer code stream, which is not limited in this application) is received. The first code stream includes, but is not limited to: first basic image coding data, first enhanced image coding data, and first metadata coding data. Optionally, the metadata coding data may also be transmitted via a separate metadata code stream, which is not limited in this application.

[0162] The decapsulation module (which can also be regarded as one module with the encapsulation module) is used to decapsulate the first code stream and output first basic image coding data, first enhanced image coding data, and first metadata coding data.

[0163] The decoding module (also referred to as the image decoding module, which is not limited in this application) is used to decode the first basic image coded data to obtain the first basic image, decode the first enhanced image coded data to obtain the first enhanced image, and decode the first metadata coded data to obtain the first metadata.

[0164] In the embodiment of the present application, each code stream includes at least one image (also referred to as image data). For example, after decoding the enhanced image code stream, at least one enhanced image (enhance1...enhanceN) can be obtained, and after decoding the basic image code stream, at least one basic image (base1...baseN) can be obtained. In the embodiment of the present application, the collection of at least one enhanced image and at least one basic image is represented by the abbreviation image data, which will not be repeated below.

[0165] Optionally, the decoding module may decode the code stream by using a decoder such as High Efficiency Video Coding (HEVC), Joint Photographic Experts Group JPEG, etc.

[0166] In a possible implementation, the image data (including enhanced images and basic images) acquired by the decoding module can be image data in any color space form such as RGB or YUV, which is not limited in this application. The color space form of the enhanced image and the basic image can be the same or different, which is not limited in this application.

[0167] In another possible implementation, the bit width of the image data may be 8 bits, 10 bits, 12 bits, etc., which is not limited in this application.

[0168] In an embodiment of the present application, metadata can be embedded into a user-defined part of the bitstream during encoding. For example, the custom part can be a Supplemental Enhancement Information (SEI) field in HEVC or Versatile Video Coding (VVC), or a custom Network Abstract Layer (NAL) unit, or other reserved fields. The custom part can also be a JEIF-encapsulated APP extended information field or an MP4-encapsulated data segment, etc., which can be set according to actual needs and is not limited by this application.

[0169] In the embodiment of the present application, the metadata includes but is not limited to at least one of the following: data format of the source, region division information, region traversal order information, image features, tone mapping information and other data.

[0170] In the embodiment of the present application, the tone mapping information includes but is not limited to at least one of the following: tone mapping parameters, target image processing methods, and adjustment parameters. Among them, the tone mapping parameters include but are not limited to at least one of the following: basic image tone mapping parameters and enhanced image tone mapping parameters. The specific usage will be described in detail below.

[0171] Please continue to refer to Fig. 7A , an image processing module is used to perform image processing operations on the first basic image and the first enhanced image, and output a second basic image and a second enhanced image after image processing.

[0172] The encoding module is used to encode the second basic image, the second enhanced image and the first metadata to obtain the encoded basic image (for example, recorded as the second basic image encoding data, or the third encoding data), the enhanced image (for example, recorded as the second enhanced image encoding data, or the fourth encoding data) and the metadata (for example, recorded as the second metadata encoding data).

[0173] The encapsulation module is used to encapsulate the encoded basic image, enhanced image and metadata and output a second code stream. The second code stream includes second basic image encoded data, second enhanced image encoded data and second metadata encoded data.

[0174] Figure 7B As a system block diagram of another image processing system shown as an example, the transcoding end (which may also be called a transcoding device, a transcoding equipment, a transcoding module, a transcoding server, etc., which is not limited in this application) may optionally include but is not limited to: a decapsulation module, a decoding module, an enhanced image generation module, an image processing module, an encoding module and an encapsulation module, etc.

[0175] At the transcoding end, a first code stream (also referred to as a first double-layer code stream, which is not limited in this application) is received. The first code stream includes but is not limited to: first basic image coding data, first enhanced image coding data, and first metadata coding data. Optionally, the metadata may also be transmitted via a separate metadata code stream, which is not limited in this application.

[0176] The decapsulation module (which can also be regarded as one module with the encapsulation module) is used to decapsulate the first code stream and output first basic image coding data, first enhanced image coding data, and metadata coding data.

[0177] The decoding module (also referred to as the image decoding module, which is not limited in this application) is used to decode the first basic image coded data to obtain the first basic image, decode the first enhanced image coded data to obtain the first enhanced image, and decode the metadata coded data to obtain metadata.

[0178] In the embodiment of the present application, each code stream includes at least one image (also referred to as image data). For example, after decoding the enhanced image coded data, at least one enhanced image (enhance1...enhanceN) can be obtained, and after decoding the basic image coded data, at least one basic image (base1...baseN) can be obtained. In the embodiment of the present application, the collection of at least one enhanced image and at least one basic image is represented by the abbreviation image data, which will not be repeated below.

[0179] Optionally, the decoding module may decode the code stream by using a decoder such as High Efficiency Video Coding (HEVC), Joint Photographic Experts Group JPEG, etc.

[0180] In a possible implementation, the image data (including enhanced images and basic images) acquired by the decoding module can be image data in any color space form such as RGB or YUV, which is not limited in this application. The color space form of the enhanced image and the basic image can be the same or different, which is not limited in this application.

[0181] In another possible implementation, the bit width of the image data may be 8 bits, 10 bits, 12 bits, etc., which is not limited in this application.

[0182] In an embodiment of the present application, metadata can be embedded into a user-defined part of the bitstream during encoding. For example, the custom part can be a Supplemental Enhancement Information (SEI) field in HEVC or Versatile Video Coding (VVC), or a custom Network Abstract Layer (NAL) unit, or other reserved fields. The custom part can also be a JEIF-encapsulated APP extended information field or an MP4-encapsulated data segment, etc., which can be set according to actual needs and is not limited by this application.

[0183] In the embodiment of the present application, the metadata includes but is not limited to at least one of the following: data format of the source, region division information, region traversal order information, image features, tone mapping information and other data.

[0184] In the embodiment of the present application, the tone mapping information includes but is not limited to at least one of the following: tone mapping parameters, target image processing methods, and adjustment parameters. Among them, the tone mapping parameters include but are not limited to at least one of the following: basic image tone mapping parameters and enhanced image tone mapping parameters. The specific usage will be described in detail below.

[0185] Please continue to refer to Figure 7B The enhanced image generation module is used to obtain a composite image (referred to as the first composite image in the present embodiment of the application) based on the basic image (i.e., the first basic image) and the enhanced image (i.e., the first enhanced image). The enhanced image generation module inputs the basic image and the composite image into the image processing module.

[0186] The image processing module is used to perform image processing on the image data and output the processed image data. The image data includes at least one of a basic image, an enhanced image, and a synthesized image. In this example, the basic image and the synthesized image are inputs of the image processing module. During the process of the image processing module processing the image, the image processing module can perform image processing on the basic image and / or the synthesized image.

[0187] Optionally, the image processing module performs image processing on a basic image (for example, which may be recorded as a first basic image in an embodiment of the present application) and a composite image (for example, which may be recorded as a first composite image in an embodiment of the present application), and outputs the processed composite image (which may be recorded as a second composite image in an embodiment of the present application) and a basic image (for example, which may be recorded as a second basic image in an embodiment of the present application).

[0188] Optionally, the image processing module may determine a target processing method and perform image processing on the image data according to the target processing method.

[0189] The enhanced image generation module is used to obtain an enhanced image (which may be recorded as the second enhanced image in the embodiment of the present application) based on the basic image (ie, the second basic image) and the synthesized image (ie, the second synthesized image).

[0190] The encoding module is used to encode a basic image (i.e., a second basic image), an enhanced image (i.e., a second enhanced image), and metadata, and output the encoded basic image (for example, in the embodiment of the present application, recorded as the second basic image encoding data, or the third encoding data, used to represent the second basic image), the enhanced image (for example, in the embodiment of the present application, recorded as the second enhanced image encoding data, or the fourth encoding data, used to represent the second enhanced image) and metadata.

[0191] The encapsulation module is used to output a second code stream (also referred to as a second double-layer code stream, which is not limited in this application) according to the encoded basic image, enhanced image and metadata.

[0192] Optionally, the forwarding end may further include a communication module (or may be called a transceiver module) for receiving the first code stream or sending the second code stream.

[0193] Optionally, the forwarding section may further include a storage module for storing the second code stream.

[0194] Exemplarily, the receiving end receives the second code stream, and performs operations such as decapsulation and decoding on the second code stream to obtain a second basic image and a second enhanced image. The receiving end may generate a second composite image based on the second basic image and the second enhanced image. Specifically, the receiving end may merge the second basic image and the second enhanced image according to the metadata to obtain a second composite image. The second composite image generated by the receiving end is Figure 7B The second composite image output by the image processing module in is the same. It can be understood that the second basic image and the second enhanced image transcoded by the transcoding end can be used to synthesize the second composite image. Optionally, in some instances, the transcoding end may directly process the first basic image and the first enhanced image during the image processing process to obtain the second basic image and the second enhanced image, that is, in this example, the transcoding end does not generate the first composite image and the second composite image. In this example, the first basic image and the first enhanced image can still be understood as being used to synthesize the first composite image. Similarly, the second basic image and the second enhanced image are used to synthesize the second composite image.

[0195] Figure 8 For a flow chart of an exemplary image processing method, please refer to Figure 8 , including but not limited to the following steps:

[0196] S801, obtaining a first double-layer code stream, the first double-layer code stream comprising first coded data and second coded data, the first coded data being used to represent a first basic image, the second coded data being used to represent a first enhanced image, the first basic image and the first enhanced image being used to synthesize a first synthesized image.

[0197] Exemplarily, the transcoding end obtains a first code stream. In an embodiment of the present application, the first code stream is a double-layer code stream, including encoding data of a basic image (i.e., first encoding data) and encoding data of an enhanced image (i.e., second encoding data). To distinguish the basic image and enhanced image from the subsequent basic image, the basic image in the first code stream is recorded as a first basic image, and the enhanced image is recorded as a first enhanced image. Optionally, the first basic image may include one or more basic images. The first enhanced image may include one or more enhanced images.

[0198] Optionally, the first code stream includes metadata encoding data. In other embodiments, the metadata may also be a separate code stream, which is not limited in this application.

[0199] Exemplarily, the decapsulation module decapsulates the first code stream and outputs the first enhanced image coded data, the first basic image coded data and the metadata coded data. Then, the decoding module decodes each code stream according to the standard corresponding to the code stream format and outputs the image data (including the first enhanced image and the first basic image) and the metadata.

[0200] The decoding module decodes the encoded data to obtain a first basic image and a first enhanced image.

[0201] In the embodiment of the present application, the first basic image and the first enhanced image may be used to synthesize a first synthesized image (which may be recorded as HDRPic). The specific synthesis method will be described in detail below.

[0202] S802, performing image processing operations based on the first double-layer code stream to obtain third coded data and fourth coded data, wherein the third coded data is used to represent the second basic image, and the fourth coded data is used to represent the second enhanced image. The second basic image and the second enhanced image are used to synthesize a second synthesized image, and there are differences associated with the image processing operation between the second synthesized image and the first synthesized image.

[0203] Exemplarily, the enhanced image generation module performs tone mapping processing on at least one first basic image and at least one first enhanced image based on the tone mapping parameters in the metadata, and then merges the at least one first basic image and the at least one first enhanced image after the tone mapping processing to obtain a first composite image.

[0204] The tone mapping processing of the first basic image and the first enhanced image is described in detail below.

[0205] 1) According to the tone mapping parameters of the basic image, the basic image is tone mapped to obtain a tone mapped basic image (denoted as baseAfter).

[0206] Fig. 9 This is a schematic diagram of an exemplary tone mapping process. Please refer to Fig. 9 , the enhanced image generation module performs tone mapping processing on at least one basic image (base_1...base_N) based on the basic image tone mapping parameters to obtain at least one tone mapped basic image (denoted as baseAfter_1...baseAfter_N). It should be noted that in this step, only the processing process of one basic image (such as base_1) is used as an example for description, and the same steps are used for the processing of other basic images, which will not be described one by one in this application.

[0207] Exemplarily, tone mapping processing methods of the basic image include but are not limited to the following methods:

[0208] Method 1:

[0209] baseAfter[i]=Base[i], where i is a pixel in the image.

[0210] In this method, the basic image is the tone mapping image, which can also be understood as no tone mapping processing is performed.

[0211] Method 2:

[0212] baseAfter[i]=base[i]*THH+(A-base[i])*THL

[0213] Among them, THH is the upper limit value of the basic image, and THL is the lower limit value of the basic image. It can be understood that THH and THL are used to indicate the value range of the basic image. A is the maximum value stored in the basic image. It can be understood that the second method is to normalize the basic image so that its value is normalized to the value range indicated by THH and THL. Among them, when the basic image base is normalized to between 0 and 1.0, the value of A is 1.0.

[0214] Optionally, THH and THL are tone mapping parameters, that is, they are included in the tone mapping information of the metadata. Optionally, A is a value pre-stored by the decoding module, that is, the encoding end encodes based on the A value, and correspondingly, the transcoding end also performs tone mapping processing based on the A value. The specific value can be set according to actual needs and is not limited in this application.

[0215] In some examples, THH and THL may also be pre-stored in the decoding module, that is, the agreed values ​​between the transcoding end and the encoding end, which may be set according to actual needs and are not limited in this application.

[0216] Method 3:

[0217] baseAfter[i]=base[i]+THL

[0218] Wherein, THL is the lower limit of the basic image. It can be understood that THL is used to indicate the value range of the basic image, that is, the range not lower than THL. It can be understood that the third method is to normalize the basic image so that its value is normalized to a value range not lower than that indicated by THL.

[0219] Optionally, THL is a tone mapping parameter, that is, included in the tone mapping information of the metadata. In some examples, THL may also be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0220] Method 4:

[0221] baseAfter[i]=THH+A-base[i]

[0222] Among them, THH is the upper limit of the basic image. It can be understood that THH is used to indicate the value range of the basic image, that is, the range that does not exceed (and is higher than) THH. A is the maximum value stored in the basic image. It can be understood that the second method is to normalize the basic image so that its value is normalized to a value range not higher than that indicated by THH. Among them, when the basic image base is normalized to between 0 and 1.0, the value of A is 1.0.

[0223] Optionally, THH is a tone mapping parameter, that is, included in the tone mapping information of the metadata. In some examples, THH may also be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0224] Method 5:

[0225] baseAfter[i]=TMB(base[i])

[0226] Wherein, TMB() is a tone mapping parameter. Optionally, TMB() can be a global and or local tone mapping function. In this manner, the decoding module can perform global tone mapping processing or local tone mapping processing on the base based on the TMB() function.

[0227] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TMB() function for expressing a mapping relationship, and the decoding function may determine the corresponding TMB() function based on these parameters.

[0228] In some examples, TMB() may also be pre-stored in the decoding module, that is, the agreed value between the transcoding end and the encoding end.

[0229] Method 6:

[0230] baseAfter[i]=TMB()*base[i]

[0231] Wherein, TMB() is a tone mapping parameter. Optionally, TMB() can be a global and or local tone mapping function. In this manner, the decoding module can perform global tone mapping processing or local tone mapping processing on the base based on the TMB() function.

[0232] In this way, tone mapping processing based on the tone mapping function can make the synthesized image closer to the original synthesized image (i.e., the synthesized image generated by the transmitter) when the synthesized image is subsequently obtained based on baseAfter, so that the data volume of the enhanced image in the second code stream (the concept can be referred to below) is smaller.

[0233] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TMB() function for expressing a mapping relationship, and the decoding function may determine the corresponding TMB() function based on these parameters.

[0234] In some examples, TMB() may also be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0235] Method 7:

[0236] baseAfter[i]=TMB(TMB1(base[i]))

[0237] Among them, TMB() and TMB1() are both tone mapping parameters. Optionally, TMB() can be a global and or local tone mapping function. Optionally, TMB1() can be in the form of linear, spline, segmented curve, etc. The role of TMB1() can be understood as being used to indicate that the basic image base[i] is transformed into a specified range. In other words, the tone mapping parameters in Methods 2 to 4 (such as THL or THH) can also be expressed by TMB1(), which can be understood as the decoding module can obtain the corresponding TMB1() based on the tone mapping parameters (such as THH, etc.). Of course, TMB1() in Method 7 (and Method 8) can also be expressed in any of Methods 2 to 4, and this application is not limited to this.

[0238] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TMB() and TMB1() functions for expressing mapping relationships, and the decoding function may determine the corresponding TMB() and TMB1() functions based on these parameters.

[0239] In some examples, TMB() and / or TMB1() may also be pre-stored in the enhanced image generation module, that is, the agreed values ​​between the transcoding end and the encoding end.

[0240] Method 8:

[0241] baseAfter[i]=TMB()*TMB1(base[i])

[0242] Among them, TMB() and TMB1() are both tone mapping parameters. Optionally, TMB() can be a global and or local tone mapping function. Optionally, TMB1() can be in the form of linear, spline, segmented curve, etc. The role of TMB1() can be understood as being used to indicate that the basic image base[i] is transformed into a specified range. In other words, the tone mapping parameters in Methods 2 to 4 (such as THL or THH) can also be expressed by TMB1(), which can be understood as the decoding module can obtain the corresponding TMB1() based on the tone mapping parameters (such as THH, etc.). Of course, TMB1() in Method 7 (and Method 8) can also be expressed in any of Methods 2 to 4, and this application is not limited to this.

[0243] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TMB() and TMB1() functions for expressing mapping relationships, and the decoding function may determine the corresponding TMB() and TMB1() functions based on these parameters.

[0244] In some examples, TMB() and / or TMB1() may also be pre-stored in the enhanced image generation module, that is, the agreed values ​​between the transcoding end and the encoding end.

[0245] Method Nine:

[0246] baseAfter[i]=TMB()*TMB1(baseMid1[i]);

[0247] or,

[0248] baseAfter[i]=TMB(TMB1(baseMid1[i]));

[0249] in,

[0250] baseMid1[i]=F[base[i]] or ∑F[n]*base[i+n], baseAfter2[i]=base[i]-baseMid1[i]

[0251] Among them, the tone mapping parameters include but are not limited to: TMB(), TMB1() and F[]. Optionally, TMB() can be a global and or local tone mapping function. Optionally, TMB1() can be in the form of linear, spline, segmented curve, etc. The role of TMB1() can be understood as being used to indicate that the basic image base[i] is transformed into a specified range. In other words, the tone mapping parameters in Methods 2 to 4 (such as THL or THH) can also be expressed by TMB1(), which can be understood as the decoding module can obtain the corresponding TMB1() based on the tone mapping parameters (such as THH, etc.). Of course, TMB1() in Method 7 (and Method 8) can also be expressed in any of Methods 2 to 4, and this application is not limited to this.

[0252] Optionally, F[] is a spatial filter parameter or other image smoothing function. The enhanced image generation module can obtain baseMid1[i] and baseAfter2[i] based on F[]. There are various filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not limited in this application.

[0253] Optionally, the tone mapping parameters of the metadata may also include some parameters in TMB(), TMB1() and F[] functions for expressing mapping relationships, and the decoding function may determine the corresponding TMB(), TMB1() and F[] functions based on these parameters.

[0254] In some examples, at least one of TMB(), TMB1() and F[] may also be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0255] Method 10:

[0256] baseAfter[i]=F[base[i]] or ∑F[n]*base[i+n], baseAfter2[i]=base[i]-baseMid1[i]

[0257] Wherein, F[] is a spatial filter parameter or other image smoothing function. The enhanced image generation module can obtain baseMid1[i] and baseAfter2[i] based on F[]. There are various forms of filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not limited in this application.

[0258] Optionally, the tone mapping parameters of the metadata may also include some parameters in the F[] function for expressing the mapping relationship, and the decoding function may determine the corresponding F[] function based on these parameters.

[0259] In some examples, F[] may be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0260] 2) According to the enhanced image tone mapping parameters, tone mapping is performed on the enhanced image to obtain a tone mapped enhanced image (denoted as enhanceAfter).

[0261] For details, please refer to Fig. 9 , the enhanced image generation module performs tone mapping processing on at least one first enhanced image (enhance_1...enhance_N) based on the enhanced image tone mapping parameters to obtain at least one tone mapped enhanced image (recorded as enhanceAfter_1...enhanceAfter_N). It should be noted that in this step, only the processing process of one enhanced image (such as enhance_1) is used as an example for description, and the same steps are used for the processing of other enhanced images, which will not be described one by one in this application.

[0262] Exemplary tone mapping processing methods for enhancing images include but are not limited to the following methods:

[0263] Method 1:

[0264] enchenceAfter[i]=enchence[i], where i is a pixel in the image.

[0265] In this method, the enhanced image is the tone-mapped image, which can also be understood as no tone mapping processing is performed.

[0266] Method 2:

[0267] enhanceAfter[i]=enhance[i]*THH+(A-enhance[i])*THL

[0268] Among them, THH is the upper limit value of the enhanced image, and THL is the lower limit value of the enhanced image. It can be understood that THH and THL are used to indicate the numerical range of the enhanced image. A is the maximum value stored in the enhanced image. It can be understood that the second method is to normalize the enhanced image so that its value is normalized to the numerical range indicated by THH and THL. Among them, when the enhanced image enhance is normalized to between 0 and 1.0, the value of A is 1.0.

[0269] Optionally, THH and THL are tone mapping parameters, that is, they are included in the tone mapping information of the metadata. Optionally, A is a value pre-stored by the decoding module, that is, the encoding end encodes based on the A value, and correspondingly, the transcoding end also performs tone mapping processing based on the A value. The specific value can be set according to actual needs and is not limited in this application.

[0270] In some examples, THH and THL may also be pre-stored in the decoding module, that is, the agreed values ​​between the transcoding end and the encoding end, which may be set according to actual needs and are not limited in this application.

[0271] Method 3:

[0272] enhanceAfter[i]=enhance[i]+THL

[0273] Wherein, THL is the lower limit of the enhanced image. It can be understood that THL is used to indicate the numerical range of the enhanced image, that is, the range not lower than THL. It can be understood that the third method is to normalize the enhanced image so that its value is normalized to a numerical range not lower than that indicated by THL.

[0274] Optionally, THL is a tone mapping parameter, that is, included in the tone mapping information of the metadata. In some examples, THL may also be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0275] Method 4:

[0276] enhanceAfter[i]=THH+A-enhance[i]

[0277] Among them, THH is the upper limit value of the enhanced image. It can be understood that THH is used to indicate the numerical range of the enhanced image, that is, the range not exceeding (and higher than) THH. A is the maximum value stored in the enhanced image. It can be understood that the second method is to normalize the enhanced image so that its value is normalized to a numerical range not higher than that indicated by THH. Among them, when the enhanced image enhance is normalized to between 0 and 1.0, the value of A is 1.0.

[0278] Optionally, THH is a tone mapping parameter, that is, included in the tone mapping information of the metadata. In some examples, THH may also be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0279] Method 5:

[0280] enhanceAfter[i]=TME(enhance[i])

[0281] Wherein, TME() is a tone mapping parameter. Optionally, TME() can be a global and or local tone mapping function. In this manner, the decoding module can perform global tone mapping processing or local tone mapping processing on the enhanced image based on the TME() function.

[0282] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TME() function for expressing a mapping relationship, and the decoding function may determine the corresponding TME() function based on these parameters.

[0283] In some examples, TME() may also be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0284] 3) Obtain an intermediate image (denoted as recHDR) according to the tone-mapped base image layer data (baseAfter) and the tone-mapped enhanced image layer data (enhanceAfter).

[0285] For details, please refer to Fig. 9 As shown, the decoding module obtains at least one intermediate image (recHDR_1...recHDR_N) based on at least one tone-mapped basic image (baseAfter_1...baseAfter_N) and at least one tone-mapped enhanced image (enhanceAfter_1...enhanceAfter_N). In the embodiment of the present application, it can also be understood that the tone-mapped basic image and the enhanced image are merged to obtain the corresponding intermediate image. It should be noted that this step is only described by taking the generation of a single intermediate image as an example, and this application will not explain them one by one.

[0286] Exemplarily, the intermediate image is obtained in the following ways, including but not limited to:

[0287] Method 1:

[0288] recHDR[i]=baseAfter[i]*enhanceAfter[i]

[0289] Method 2:

[0290] recHDR[i]=baseAfter[i]+enhanceAfter[i]

[0291] Method 3:

[0292] recHDR[i]=f1 baseAfter[i]+f2(enhanceAfter[i]).

[0293] Method 4:

[0294] recHDR[i]=f1 baseAfter[i]*f2(enhanceAfter[i]).

[0295] Wherein, f1() and f2() are conversion functions in the numerical domain. Exemplarily, the enhanced image generation module may transform the tone mapping enhanced image and / or the tone mapping basic image into the same numerical domain based on the f() function and then merge (including adding or multiplying) them.

[0296] It should be noted that the merging methods involved in the embodiments of the present application are only illustrative examples. In other embodiments, other feasible merging methods may also be adopted, and the present application does not limit them.

[0297] Exemplarily, the tone mapping information of the metadata may include information for indicating any merging scheme, and the enhanced image generation module may select a corresponding method based on the indication of the metadata.

[0298] In some instances, the merging scheme may also be pre-set in the enhanced image generation module, that is, for the transcoding end and the encoding end, which is not limited in this application.

[0299] In a possible implementation, as described above, the decoded enhanced image and basic image may be one or more. In the case where there are multiple enhanced images and / or basic images, the enhanced image generation module performs the above processing on the multiple images to obtain recHDR[i], and the specific processing method is as follows:

[0300] Method 1:

[0301] recHDR[i]=A*g(baseAfter1[i]*f1(enchenceAfter1[i]))+B*g(baseAfter2[i]*f2(enchenceAfter2[i]))+C*g(baseAfter3[i]*f3(enchenceAfter3[i]))…

[0302] Among them, f1(), f2() and f3() are conversion functions. The above functions may be included in the tone mapping information in the metadata, or may be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or may be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0303] Method 2:

[0304] recHDR[i]=A*g(baseAfter1[i]*f1(enchenceAfter1[i]))+B*g(baseAfter2[i])+C*g(baseAfter3[i])……,

[0305] Among them, f1() is a conversion function. The above function may be included in the tone mapping information in the metadata, or it may be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or it may be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0306] Exemplarily, this method can be used to process a scene in which a code stream includes one enhanced image and multiple basic images.

[0307] Method 3:

[0308] recHDR[i]=A*g(baseAfter1[i]+f1(enchenceAfter1[i]))+B*g(baseAfter2[i]+f2(enchenceAfter2[i]))+C*g(baseAfter3[i]+f3(enchenceAfter3[i]))…

[0309] Among them, f1(), f2() and f3() are conversion functions. The above functions may be included in the tone mapping information in the metadata, or may be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or may be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0310] Method 4:

[0311] recHDR[i]=A*g(baseAfter1[i]+f1(enchenceAfter1[i]))+B*g(baseAfter2[i])+C*g(baseAfter3[i])……

[0312] Among them, f1() is a conversion function. The above function may be included in the tone mapping information in the metadata, or it may be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or it may be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0313] Exemplarily, this method can be used to process a scene in which a code stream includes one enhanced image and multiple basic images.

[0314] Method 5:

[0315] recHDR[i]=A*baseAfter1[i]+A1*f1(enchenceAfter1[i])+baseAfter1[i]+B*baseAfter2[i]+B1*f1(enchenceAfter2[i])+C*baseAfter3[i]+C1*f1(enchenceAfter3[i])……

[0316] Among them, f1(), f2() and f3() are conversion functions. The above functions may be included in the tone mapping information in the metadata, or may be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or may be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0317] 4) Based on the intermediate image, obtain the composite image (referred to as HDRPic)

[0318] Specifically, as described above, the tone mapping information of the metadata may further include adjustment parameters. The enhanced image generation module may perform numerical adjustment on at least one intermediate image (recHDR_1 . . . recHDR_N) based on the adjustment parameters to obtain a composite image.

[0319] The adjustment method is as follows:

[0320] Method 1:

[0321] HDRPic[i]=recHDR[i]

[0322] In this manner, it can also be understood that no numerical adjustment is made, that is, the numerical adjustment amplitude is 0.

[0323] Method 2:

[0324] HDRPic[i]=recHDR[i]+X

[0325] Among them, X is the adjustment parameter.

[0326] Optionally, X may be an adjustment parameter in the tone mapping information of the metadata, or may be preset by the transcoding end, which is not limited in this application.

[0327] Method 3:

[0328] HDRPic[i]=g(recHDR[i])

[0329] Here, g() is an inverse normalization function, a mapping function, a tone mapping function or an inverse mapping function.

[0330] Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or may be preset by the transcoding end, which is not limited in this application, and the decoding function may determine the corresponding g() function based on these parameters. Optionally, g() may be a global and or local tone mapping function. In this manner, the enhanced image generation module may perform global tone mapping processing or local tone mapping processing on the enhanced image based on the g() function.

[0331] It should be noted that the numerical adjustment step is an optional step. If the encoding end performs numerical adjustment during the encoding process, the transcoding end will perform numerical adjustment accordingly. Conversely, if the encoding end does not perform numerical adjustment during the encoding process, the transcoding end does not need to perform numerical adjustment. Of course, it can also be understood that the numerical adjustment is performed, and the numerical adjustment amplitude is 0 (that is, the first method of numerical adjustment).

[0332] It should be noted that this application does not limit baseAfter[i] and enchenceAfter[i] to any domain, which can be a linear domain or a PQ domain, a LOG domain, etc. This application also does not limit the color space of baseAfter[i] and enchenceAfter[i], which can be a YUV, RGB, Lab, HSV, etc. color space.

[0333] It should be further explained that after the transcoding end obtains the composite image, it can further perform other image processing on the composite image, which is not limited in this application.

[0334] Exemplarily, the enhanced image generation module outputs the first composite image and the first basic image to the image processing module. The image processing module may perform image processing on the first composite image and the first basic image respectively according to the target image processing method to obtain a composite image (for example, recorded as the second composite image in the embodiment of the present application) and a basic image (for example, recorded as the second basic image in the embodiment of the present application) after image processing.

[0335] During the processing by the image processing module, the image processing module needs to determine the target image processing method. In the embodiment of the present application, the method in which the image processing module determines the target image processing method may include but is not limited to any of the following:

[0336] 1) Determined by user settings

[0337] 2) Determine based on metadata

[0338] 3) Determined according to the preset method.

[0339] In the embodiment of the present application, a plurality of image processing methods are provided (which will be described in detail below). The specific method selected as the target image processing method can be selected through any one of the above three methods.

[0340] In one example, during the transcoding process, the transcoding end can provide different options, each of which can correspond to one or more image processing methods (the corresponding relationship can be set according to actual needs, and this application does not limit it) for the user to choose. The user can select the corresponding image processing method according to the required image quality, thereby achieving controllability of the image quality after transcoding. Optionally, the user described in the embodiment of the present application can refer to an end user, or a developer or operation and maintenance personnel, etc., which is not limited by this application.

[0341] In the embodiment of the present application, the selectable image processing operations include but are not limited to at least one of the following: image scaling, image sharpening, image blunting, adding watermarks, removing watermarks, etc.

[0342] Exemplarily, image scaling includes image magnification and image reduction. Image reduction can optionally be to reduce the image size (i.e., resolution) so that the size of the reduced image file becomes smaller than before processing, effectively saving the storage cost of the transcoding end and the receiving end, and reducing the transmission overhead. For example, if user 1 shares a double-layer picture to a third-party platform, the third-party platform provides transcoding services. Considering the storage cost, the platform may downsample the image (reduce the resolution) before transmitting it to user 2. Conversely, image magnification can optionally be to enlarge the image size (i.e., resolution), and the image file size becomes larger than before processing, which can improve the image quality and further improve the image display effect. For example, after user 1 takes a double-layer image, he drags it into the image editing software (i.e., the transcoding end) for processing. The transcoding end can transcode the double-layer image, and perform image processing on the image data during the transcoding process to enlarge the image data and improve the image resolution.

[0343] Exemplarily, image sharpening increases the contrast between edge pixels and emphasizes the transition between light and dark areas. This makes the image outline clearer, enhances local contrast, and shows more details in the image. For example, the transcoding end receives a double-layer code stream, and the image in the code stream may not be clear when it is shot because the original image is not imported. The user can select the sharpening function provided by the transcoding end, and the transcoding end can sharpen the image to improve the image quality. Conversely, image passivation can optionally reduce the contrast between edge pixels to make the image outline more blurred.

[0344] There are two main purposes for adding watermarks. One is user demand, and the other is the behavior of the transcoding platform based on publicity or security considerations. The effect is to add some new logos or content to the image. The transcoding end can add the corresponding specified watermark to the image during the image processing process. Conversely, watermark removal can be optionally to remove the watermark in the image.

[0345] In another example, as described above, the tone mapping information of the metadata may include a target processing method. That is, the image processing module may determine the image processing method based on the target processing method carried in the tone mapping information of the metadata.

[0346] In another example, the transcoding end may preset a target image processing method.

[0347] Exemplarily, the image processing module performs an image processing operation on the first composite image and the first basic image based on a target image processing method to obtain a second composite image and a second basic image.

[0348] The present application provides the following image processing methods. The image processing module can perform image processing operations on the first composite image and the first basic image based on the following image processing methods to obtain a third HDR image and a second basic image.

[0349] The target image processing method is one of the following image processing methods. It should be noted that the image processing methods listed in the embodiments of the present application are only for illustrative purposes, and other embodiments may include more image processing methods, which are not limited in the present application.

[0350] Method 1: Perform an image processing operation on the first composite image (for example, this application may record it as the first image processing operation), and perform an image processing operation on the first basic image (for example, this application may record it as the second image processing operation). In order to clearly describe different image processing operations, the first image processing operation and the second image processing operation are distinguished below, and no repeated description is given below.

[0351] In the embodiment of the present application, the image processing operations include but are not limited to: scaling, sharpening, adding watermarks, etc. Of course, the image processing operations listed in the embodiment of the present application are only illustrative examples and can be set according to actual needs, and the present application does not limit them.

[0352] That is to say, in method one, the first image processing operation performed by the image processing module on the first composite image may include at least one of scaling, sharpening, etc. Of course, in other embodiments, other image processing methods may also be included, which can be set according to actual needs and is not limited in this application.

[0353] The image processing module performs the first image processing operation on the first composite image, which can also be understood as modifying the first composite image, that is, modifying the image. The modification includes modifying the image resolution, image color, and image content, which will not be repeated below.

[0354] Exemplarily, the image processing module outputs a second composite image.

[0355] And, the image processing module performs a second image processing operation on the first basic image and outputs a second basic image. The description may refer to the first synthesized image, which will not be repeated here.

[0356] In the embodiment of the present application, the first image processing operation and the second image processing operation may be the same. In other embodiments, the first image processing operation and the second image processing operation may also be different, which is not limited in the present application.

[0357] For example, the image processing module can reduce and sharpen the first composite image, and also reduce and sharpen the first basic image. In this way, the quality of the obtained second composite image and the second basic image is reduced, but the data volume is also reduced. In this example, the data volume of the transcoded image is reduced, which can effectively save the storage resources of the transcoding end and reduce the bandwidth occupied by the code stream during the transmission process.

[0358] Method 2: performing the first image processing operation on the first composite image, and not processing the first basic image.

[0359] Exemplarily, the image processing module performs a first image processing operation on the first composite image, which can also be understood as modifying the first composite image, and outputs a second composite image.

[0360] The image processing module does not process the first basic image, and the second basic image output by the image processing module to the encoding module is the first basic image. It can be understood that after the first basic image is input to the image processing module, the second basic image output is the first basic image.

[0361] For other parts not described, please refer to method 1 and will not be repeated here.

[0362] In a possible implementation, some simple processing may be performed on the first basic image, such as scaling, etc., which is not limited in this application.

[0363] Method three: The first composite image is not processed, and the second image processing operation is performed on the first basic image.

[0364] Exemplarily, the image processing module performs the second image processing operation on the first basic image, which can also be understood as modifying the first basic image, and outputting the second basic image.

[0365] The image processing module does not process the first composite image, and the second composite image output by the image processing module to the encoding module is the first composite image. It can be understood that after the first composite image is input to the image processing module, the second composite image output is the first composite image.

[0366] For other parts not described, please refer to method 1 and will not be repeated here.

[0367] In a possible implementation, some simple processing may be performed on the first composite image, such as scaling, etc., which is not limited in this application.

[0368] Method 4: performing a first image processing operation on the first composite image to obtain a second composite image.

[0369] Based on the second synthesized image, a second basic image is obtained.

[0370] Exemplarily, the image processing module performs a first image processing operation on the first composite image, which can also be understood as modifying the first composite image, and outputs a second composite image.

[0371] The image processing module can obtain a second basic image based on the second synthetic image. Specifically, the image processing module processes the second synthetic image based on a preset algorithm to obtain a second basic image. The preset algorithm may include, but is not limited to: a global tone mapping algorithm, a local tone mapping algorithm, or an inverse algorithm of the TMB() function (the concept of the function can be referred to above), etc., which can be set according to actual needs and is not limited in this application. In this way, the output synthetic image can be consistent in style with the basic image, avoiding the problem of backward compatibility caused by large differences in style.

[0372] Method five: performing a first image processing operation on the first basic image to obtain a second basic image.

[0373] Based on the second basic image, a second composite image is obtained.

[0374] Exemplarily, the image processing module performs the second image processing operation on the first basic image, which can also be understood as modifying the first basic image, and outputting the second basic image.

[0375] The image processing module may obtain a second composite image based on the second basic image. Specifically, the image processing module processes the second basic image based on a preset algorithm to obtain a second composite image. The preset algorithm may include but is not limited to: a global tone mapping algorithm, a local tone mapping algorithm, or a forward algorithm of a TMB() function (the concept of the function may refer to the above).

[0376] Method six: performing a first image processing operation on the first basic image to obtain a second basic image.

[0377] A second image processing is performed based on the first enhanced image to obtain a second enhanced image.

[0378] In the embodiment of the present application, the image processing module may perform a second image processing operation on the first basic image, which may also be understood as modifying the first basic image and outputting a second basic image.

[0379] Furthermore, the image processing module may perform second image processing on the decoded first enhanced image to obtain a second enhanced image. Optionally, the first image processing and the second image processing may be the same or different.

[0380] In this example, the enhanced image generation module may not participate in the transcoding process, that is, the decoding module outputs the first basic image and the first enhanced image to the image processing module, and the image processing module processes the first basic image and the first enhanced image and outputs the second basic image and the second enhanced image.

[0381] Optionally, the second image processing of the first enhanced image by the image processing module may include but is not limited to reducing or enlarging the enhanced image, etc., which is not limited in this application.

[0382] As shown in 7B, the transcoding end can obtain the second code stream based on the second composite image and the second basic image. Specifically, the image processing module obtains the second composite image and the second basic image, and outputs the second composite image and the second basic image to the enhanced image generation module.

[0383] The enhanced image generation module obtains a second enhanced image based on the second basic image and the second synthesized image. Optionally, Figure 7B The process executed by the enhanced image generation module may also be executed by the image processing module in some examples, which is not limited in this application.

[0384] The encoding module can encode the second basic image, the second enhanced image and the metadata to obtain the second basic image encoding data (i.e., the third encoding data) and the second enhanced image encoding data (i.e., the fourth encoding data) and the encoded metadata (which can also be called the second metadata encoding data). Exemplarily, the second basic image encoding data is used to characterize the second basic image, and the second enhanced image encoding data is used to characterize the second enhanced image. Optionally, the encoding method can use any encoding method such as H.265, H.264, etc., which can be set according to actual needs and is not limited in this application.

[0385] The encapsulation module may encapsulate the second basic image coded data, the second enhanced image coded data, and the metadata coded data to obtain a second code stream, which may also be referred to as a second double-layer code stream. The second code stream includes, but is not limited to: the second basic image coded data, the second enhanced image coded data, and the metadata coded data (which may also be recorded as the second metadata coded data). Optionally, the metadata coded data may also be transmitted via a separate code stream, which is not limited in this application.

[0386] In this way, there is a difference corresponding to the image processing operation between the second synthesized image synthesized by the second basic image and the second enhanced image in the second double-layer code stream and the first synthesized image. For example, if the image processing operation is to scale the image data, the difference between the first synthesized image and the second synthesized image is the difference in resolution. If the image processing operation is to sharpen or blunt the image data, the difference between the first synthesized image and the second synthesized image is the difference in clarity, for example, the second synthesized image has higher image quality and clearer details than the first synthesized image. If the image processing operation is to add a watermark or remove a watermark, the difference between the first synthesized image and the second synthesized image can be optionally that the second synthesized image has more or less watermark than the first synthesized image.

[0387] This application provides three encoding methods:

[0388] Method 1:

[0389] 1. Encode the second basic image to obtain second basic image encoded data. If there are multiple second basic images, encode each second basic image to obtain multiple second basic images.

[0390] 2. Encode the second enhanced image to obtain second enhanced image encoded data. If there are multiple second enhanced images, encode the multiple second enhanced images to obtain multiple second enhanced images. In some embodiments, a preset encoding condition (which can be set according to actual needs) can be set, and multiple second enhanced images that meet the preset encoding condition can be encoded to obtain at least one second enhanced image, and other enhanced images that do not meet the preset encoding condition can be not processed.

[0391] Method 2:

[0392] 1. Encode the second basic image to obtain second basic image encoded data.

[0393] 2. Decode the second basic image coded data to obtain a decoded basic image.

[0394] 3. Obtain the difference (or gain) between each pixel of the decoded basic image and the second synthesized image.

[0395] 4. Encode the difference value to obtain second enhanced image encoding data.

[0396] Method 3:

[0397] 1. Encode the second basic image to obtain second basic image encoded data.

[0398] 2. Obtain the difference (or gain) between the second basic image coded data and each pixel of the second synthesized image.

[0399] 3. Encode the difference value to obtain second enhanced image encoding data.

[0400] S803, output a second double-layer code stream, where the second double-layer code stream includes third coded data and fourth coded data.

[0401] Exemplarily, the transcoding end outputs a second double-layer code stream. The second double-layer code stream includes basic image coding data (i.e., third coding data) and enhanced image coding data (i.e., fourth coding data). The transcoding end can output the second double-layer code stream to the receiving end for display, or output it to a storage device (which can be local storage or independent storage) for storage, which is not limited in this application.

[0402] The present application also provides an image processing device 1000, see Fig.10 , Fig.10 An example diagram of an image processing device provided in an embodiment of the present application is used to implement the method in an embodiment of the present application.

[0403] like Fig.10 As shown, the image processing device 1000 may include a processor 1001 for executing a program or instruction stored in a memory 1002. When the program or instruction stored in the memory 1002 is executed, the processor 1001 is used to execute Figure 8 The image encoding method in the illustrated embodiment.

[0404] Optionally, the image processing apparatus 1000 may further include a communication interface 1003. Fig.10 The communication interface 1003 is indicated by a dotted line as being optional for the image processing apparatus 1000 .

[0405] Among them, the number of processors 1001, memories 1002 and communication interfaces 1003 does not constitute a limitation on the embodiments of the present application, and during specific implementation, they can be arbitrarily configured according to business requirements.

[0406] Optionally, the memory 1002 is located outside the image processing device 1000 .

[0407] Optionally, the image processing device 1000 includes the memory 1002, the memory 1002 is connected to the at least one processor 1001, and the memory 1002 stores instructions that can be executed by the at least one processor 1001. Fig.10 The dashed line indicates that the memory 1002 is optional for the image processing apparatus 1000 .

[0408] The processor 1001 and the memory 1002 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0409] The specific connection medium between the processor 1001, the memory 1002 and the communication interface 1003 is not limited in the embodiment of the present application. Fig.10 In the embodiment, the processor 1001, the memory 1002 and the communication interface 1003 are connected via a bus 1004. Fig.10 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0410] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor implemented by reading software code stored in a memory.

[0411] Exemplarily, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0412] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (directrambus RAM, DR RAM).

[0413] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0414] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0415] The present application also provides a computer-readable storage medium, including a program or an instruction. When the program or the instruction is executed on a computer, Figure 8 The method in is executed.

[0416] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0417] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0418] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0419] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0420] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image processing method, characterized in that: include: Acquire a first double-layer code stream, the first double-layer code stream includes first coded data and second coded data, the first coded data is used to represent a first basic image, the second coded data is used to represent a first enhanced image, and the first basic image and the first enhanced image are used to synthesize a first synthesized image; performing an image processing operation based on the first double-layer code stream to obtain third coded data and fourth coded data, wherein the third coded data is used to represent a second basic image, the fourth coded data is used to represent a second enhanced image, the second basic image and the second enhanced image are used to synthesize a second synthesized image, and there is a difference associated with the image processing operation between the second synthesized image and the first synthesized image; A second double-layer code stream is output, where the second double-layer code stream includes the third coded data and the fourth coded data.

2. The method according to claim 1, characterized in that The method further comprises: Decoding the first coded data to obtain the first basic image, and decoding the second coded data to obtain the first enhanced image; The first basic image and the first enhanced image are combined to obtain the first composite image.

3. The method according to claim 2, characterized in that The performing of an image processing operation based on the first double-layer code stream to obtain third coded data and fourth coded data includes: performing the image processing operation on the first composite image to obtain the second composite image; Based on the second composite image, acquiring a second basic image; The third encoded data and the fourth encoded data are obtained based on the second synthesized image and the second basic image.

4. The method according to claim 2, characterized in that: The performing of an image processing operation based on the first double-layer code stream to obtain third coded data and fourth coded data includes: performing the image processing operation on the first composite image to obtain the second composite image; Performing the image processing operation on the first basic image to obtain a second basic image; The third encoded data and the fourth encoded data are obtained based on the second synthesized image and the second basic image.

5. The method according to claim 3 or 4, characterized in that: The obtaining the third coded data and the fourth coded data based on the second synthesized image and the second basic image comprises: obtaining the second enhanced image based on the second synthesized image and the second basic image; encoding the second basic image to obtain the third encoded data; The second enhanced image is encoded to obtain the fourth encoded data.

6. The method according to claim 3 or 4, characterized in that: The obtaining the third coded data and the fourth coded data based on the second synthesized image and the second basic image comprises: encoding the second basic image to obtain the third encoded data; The fourth encoded data is obtained based on the third encoded data and the second synthesized image.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: An operation mode of the image processing operation is obtained, where the operation mode includes at least one of the following: zooming in, zooming out, sharpening, blunting, adding a watermark, or removing a watermark.

8. The method according to claim 7, characterized in that The operation mode is preset, or the operation mode of obtaining the image processing operation includes: Determine the operation mode based on the received user operation or user demand; or, The operation mode is obtained from the first dual-layer code stream.

9. An image processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 8.

11. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 8.

Images