Image processing method and apparatus

Abstract

Provided in the embodiments of the present application are an image processing method and apparatus. The method comprises: performing image processing on a first double-layer code stream, and outputting a second double-layer code stream, wherein the first double-layer code stream comprises first coded data used for representing a first basic image, and second coded data used for representing a first enhanced image; the second double-layer code stream comprises third coded data used for representing a second basic image, and fourth coded data used for representing a second enhanced image; and the first basic image and the first enhanced image are used for synthesizing a first composite image, the second basic image and the second enhanced image are used for synthesizing a second composite image, and a difference associated with an image processing operation is present between the first composite image and the second composite image. The present application can implement image processing on a double-layer code stream, such that an output double-layer code stream meets different scenario requirements.

Description

Image processing method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 15, 2023, with application number 202311037360.4 and application name “Image Processing Method and Device”. This application also claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 13, 2024, with application number 202411111044.1 and application name “Image Processing Method and Device”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of image processing, and in particular to an image processing method and apparatus. Background Art

[0003] Currently, the image processing process generally includes front-end processing (such as image generation and encoding), intermediate processing (such as transcoding and re-encoding), and back-end processing (such as decoding and display). However, current image processing processes are typically designed for single-layer streams, supporting only HDR (High Dynamic Range) or standard dynamic range (SDR) images, for example. Processing dual-layer streams has become a pressing issue.

[0004] Summary of the Invention

[0005] The present application provides an image processing method and apparatus, wherein a transcoding end can perform image processing on a double-layer code stream to output a double-layer code stream that meets scene requirements.

[0006] In a first aspect, the present application provides an image processing method. The method includes: a transcoding end obtains a first dual-layer code stream, the first dual-layer code stream including first coded data and second coded data, the first coded data being used to represent a first basic image, the second coded data being used to represent a first enhanced image, and the first basic image and the first enhanced image being used to synthesize a first composite image. The transcoding end performs an image processing operation based on the first dual-layer code stream to obtain third coded data and fourth coded data, the third coded data being used to represent a second basic image, and the fourth coded data being used to represent a second enhanced image, the second basic image and the second enhanced image being used to synthesize a second composite image, and the second composite image and the first composite image having differences associated with the image processing operation. The transcoding end outputs a second dual-layer code stream, the second dual-layer code stream including the third coded data and the fourth coded data. Thus, the image processing method provided in the embodiments of the present application can be applied to dual-layer code stream scenarios to output a dual-layer code stream that meets the requirements of different scenarios based on the received dual-layer code stream.

[0007] Exemplarily, the transcoding end may also be a transcoding device, an image processing device, etc., which is not limited in this application.

[0008] For example, the first coded data can be understood as being obtained by encoding the first base image, the second coded data as being obtained by encoding the second enhanced image, the third coded data as being obtained by encoding the second base image, and the fourth coded data as being obtained by encoding the second enhanced image. Optionally, the encoding operation on the first base image and the second enhanced image can be performed by the transmitting end.

[0009] Exemplarily, the first dual-layer code stream obtained by the transcoding end may be obtained from the transmitting end. Optionally, the transmitting end and the transcoding end may belong to the same device or may be independent devices. For example, the transmitting end may be a terminal device and the transcoding end may be a server, which is not limited in this application.

[0010] Exemplarily, the first dual-layer code stream and the second dual-layer code stream may further include metadata.

[0011] Exemplarily, the first dual-layer code stream and the second dual-layer code stream may follow the same or different coding formats.

[0012] In a possible implementation, the method further includes: the transcoding end decodes the first encoded data to obtain a first basic image, and decodes the second encoded data to obtain a first enhanced image. The transcoding end merges the first basic image and the first enhanced image to obtain a first composite image. In this way, the transcoding end can obtain the first composite image on its own side, which can also be understood as the transcoding end performing the same decoding and synthesis operations as the receiving end, or as performing the opposite operation as the sending end. That is, the sending end obtains the first enhanced image based on the first basic image and the first composite image. The receiving end obtains the first composite image based on the first basic image and the first enhanced image. In the present application, the transcoding end obtains the first composite image based on the first basic image and the first enhanced image, and processes the first composite image in the subsequent image processing process, so that after the receiving end receives the second double-layer code stream and obtains the first composite image, what can be displayed is the first composite image processed by the transcoding end.

[0013] In one possible implementation, an image processing operation is performed based on the first double-layer code stream to obtain third encoded data and fourth encoded data, including: performing an image processing operation on the first composite image to obtain a second composite image. Based on the second composite image, a second basic image is obtained. The third encoded data and the fourth encoded data are obtained based on the second composite image and the second basic image. In this way, the transcoding end of the present application can perform corresponding image processing operations based on different scene requirements to meet the requirements of different scenes for image quality or image content. Moreover, the transcoding end performs an image processing operation on the first composite image and then obtains the second basic image based on the processed second composite image, so that the second composite image, the second basic image and the second enhanced image have the same image features as the image processing operation (for example, they are all enlarged or watermarked), thereby maintaining image consistency.

[0014] In one possible implementation, image processing operations are performed based on the first double-layer code stream to obtain third encoded data and fourth encoded data, including: performing image processing operations on the first composite image to obtain a second composite image. Performing image processing operations on the first basic image to obtain a second basic image. The third encoded data and the fourth encoded data are obtained based on the second composite image and the second basic image. In this way, the transcoding end of the present application can perform corresponding image processing operations based on different scene requirements to meet the requirements of different scenes for image quality or image content. In addition, the transcoding end performs image processing operations on the first basic image and the first composite image respectively, so that the second composite image, the second basic image and the second enhanced image have the same image features as the image processing operation (for example, they are all enlarged or watermarked), thereby maintaining image consistency.

[0015] In one possible implementation, obtaining third and fourth encoded data based on the second composite image and the second base image includes: obtaining a second enhanced image based on the second composite image and the second base image; encoding the second base image to obtain the third encoded data; and encoding the second enhanced image to obtain the fourth encoded data. In this manner, obtaining a second enhanced image based on the processed second composite image and the second base image can ensure that the enhanced image has the same (or related) image features as the second base image and the second composite image. These image features are the image features associated with the image processing operation.

[0016] In one possible implementation, obtaining third coded data and fourth coded data based on the second composite image and the second basic image includes: encoding the second basic image to obtain the third coded data, and obtaining fourth coded data based on the third coded data and the second composite image.

[0017] In one possible implementation, performing an image processing operation based on the first code stream to obtain third and fourth coded data includes: performing the image processing operation on the first composite image to obtain a second composite image; the second basic image is the first basic image; and obtaining the third and fourth coded data based on the second composite image and the second basic image.

[0018] For example, the transcoding end may input the first basic image into an image processing module. The image processing module does not process the first basic image, and the second basic image output by the image processing module serves as the first basic image. Of course, in some instances, the first basic image may not be input into the image processing module, but may be directly input as the second basic image into the backend of the image processing module for subsequent operations.

[0019] In one possible implementation, performing image processing operations based on the first bitstream to obtain third and fourth encoded data includes: the second composite image being the first composite image; performing image processing operations on the first basic image to obtain the second basic image; and obtaining the third and fourth encoded data based on the second composite image and the second basic image. In this way, when performing image processing operations, the transcoding end can perform image processing operations on different objects to meet different image quality requirements for different scenarios.

[0020] For example, the transcoding end may input the first composite image into an image processing module. The image processing module does not process the first composite image, and the second composite image output by the image processing module is the first composite image. Of course, in some examples, the first composite image may not be input into the image processing module, but may be directly input as the second composite image into the backend of the image processing module for subsequent operations.

[0021] In one possible implementation, performing image processing operations based on the first bitstream to obtain third and fourth encoded data includes: performing image processing operations on the first basic image to obtain a second basic image; obtaining a second composite image based on the second basic image; and obtaining the third and fourth encoded data based on the second composite image and the second basic image. This allows the transcoder to perform image processing operations on different objects to meet varying image quality requirements for different scenarios.

[0022] In one possible implementation, performing image processing operations based on the first code stream to obtain third and fourth encoded data includes: performing image processing operations on the first basic image to obtain a second basic image; performing image processing operations on the first enhanced image to obtain a second enhanced image; and obtaining the third and fourth encoded data based on the second enhanced image and the second basic image. In this way, when performing image processing operations, the transcoding end can perform image processing operations on different objects to meet different image quality requirements for different scenarios.

[0023] In a possible implementation, obtaining third coded data and fourth coded data based on the second enhanced image and the second basic image includes: encoding the second basic image to obtain the third coded data, and encoding the second enhanced image to obtain the fourth coded data.

[0024] In one possible implementation, the operation mode of the image processing operation is obtained, and the operation mode includes at least one of the following: enlargement, reduction, sharpening, blunting, adding watermarks, or removing watermarks. In this way, the transcoding end of this application can provide different image processing modes to adapt to different scene requirements.

[0025] In one possible implementation, the operation mode is pre-set, or the operation mode of the image processing operation is obtained, including: determining the operation mode based on a received user operation or user requirement; or obtaining the operation mode from the first dual-layer codestream. In this way, the transcoder can perform image processing on the image based on the requirements of the sender, the user's requirements, and / or requirements preset by the transcoder to output a dual-layer codestream that meets the requirements of different scenarios.

[0026] In one possible implementation, the difference includes at least one of the following: the resolution of the second composite image is different from that of the first composite image; or the clarity of the second composite image is different from that of the first composite image; or the image content of the second composite image has an additional watermark or an insufficient watermark compared to the first composite image.

[0027] In a second aspect, the present application provides an image processing device, comprising: an acquisition module for acquiring a first dual-layer code stream, the first dual-layer code stream comprising first coded data and second coded data, the first coded data being used to represent a first basic image, the second coded data being used to represent a first enhanced image, the first basic image and the first enhanced image being used to synthesize a first composite image. An image processing module for performing an image processing operation based on the first dual-layer code stream to obtain third coded data and fourth coded data, the third coded data being used to represent a second basic image, the fourth coded data being used to represent a second enhanced image, the second basic image and the second enhanced image being used to synthesize a second composite image, the second composite image and the first composite image having differences associated with the image processing operation. The image processing module is further configured to output a second dual-layer code stream, the second dual-layer code stream comprising the third coded data and the fourth coded data.

[0028] In one possible implementation, the apparatus further includes: a decoding module configured to decode the first encoded data to obtain a first basic image, and decode the second encoded data to obtain a first enhanced image; and a synthesis module configured to combine the first basic image and the first enhanced image to obtain a first synthesized image.

[0029] In one possible implementation, the image processing module is specifically configured to perform image processing operations on the first composite image to obtain a second composite image; and based on the second composite image, obtain a second basic image. The apparatus further includes an encoding module configured to obtain third and fourth encoded data based on the second composite image and the second basic image.

[0030] In one possible implementation, the image processing module is specifically used to perform image processing operations on the first composite image to obtain a second composite image; perform image processing operations on the first basic image to obtain a second basic image; the device also includes an encoding module for obtaining third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0031] In one possible implementation, the image processing module is specifically configured to obtain a second enhanced image based on the second composite image and the second basic image; the apparatus further includes an encoding module configured to encode the second basic image to obtain third encoded data, and encode the second enhanced image to obtain fourth encoded data.

[0032] In a possible implementation, the encoding module is specifically configured to encode the second basic image to obtain third encoded data; and obtain fourth encoded data based on the third encoded data and the second composite image.

[0033] In one possible implementation, the image processing module is further configured to perform an image processing operation on the first composite image to obtain a second composite image. The second basic image is the first basic image. The encoding module is configured to obtain third and fourth encoded data based on the second composite image and the second basic image.

[0034] In one possible implementation, the second composite image is the first composite image, the image processing module is configured to perform image processing on the first basic image to obtain the second basic image, and the encoding module is configured to obtain third and fourth encoded data based on the second composite image and the second basic image.

[0035] In one possible implementation, the image processing module is used to perform image processing operations on the first basic image to obtain a second basic image; based on the second basic image, a second composite image is obtained; and the encoding module is used to obtain third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0036] In one possible implementation, the image processing module is configured to perform an image processing operation on the first basic image to obtain a second basic image. The image processing operation is performed on the first enhanced image to obtain a second enhanced image. The encoding module is configured to obtain third encoded data and fourth encoded data based on the second enhanced image and the second basic image.

[0037] In a possible implementation, the encoding module is configured to encode the second basic image to obtain third encoded data, and encode the second enhanced image to obtain fourth encoded data.

[0038] In a possible implementation, an operation mode of an image processing operation is obtained, where the operation mode includes at least one of the following: zooming in, zooming out, sharpening, blunting, adding a watermark, or removing a watermark.

[0039] In a possible implementation, the operation mode is preset, or the operation mode of the image processing operation is obtained, including: determining the operation mode based on a received user operation or user demand; or obtaining the operation mode from the first dual-layer code stream.

[0040] In one possible implementation, the difference includes at least one of the following: the resolution of the second composite image is different from that of the first composite image; or the clarity of the second composite image is different from that of the first composite image; or the image content of the second composite image has an additional watermark or an insufficient watermark compared to the first composite image.

[0041] In a third aspect, the present application provides an image processing device, comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the following steps: performing an image processing operation based on a first dual-layer code stream to obtain third coded data and fourth coded data, the third coded data being used to represent a second base image, the fourth coded data being used to represent a second enhanced image, the second base image and the second enhanced image being used to synthesize a second composite image, the second composite image having differences associated with the image processing operation from the first composite image; and outputting a second dual-layer code stream, the second dual-layer code stream including the third coded data and the fourth coded data.

[0042] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: decoding the first encoded data to obtain a first basic image, and decoding the second encoded data to obtain a first enhanced image; merging the first basic image and the first enhanced image to obtain a first composite image.

[0043] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: perform image processing operations on the first composite image to obtain a second composite image; obtain a second basic image based on the second composite image; and obtain third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0044] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: perform image processing operations on the first composite image to obtain a second composite image; perform image processing operations on the first basic image to obtain a second basic image; and obtain third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0045] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: obtaining a second enhanced image based on the second composite image and the second basic image; encoding the second basic image to obtain third encoded data; encoding the second enhanced image to obtain fourth encoded data.

[0046] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: encode the second basic image to obtain third encoded data; and obtain fourth encoded data based on the third encoded data and the second composite image.

[0047] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: performing image processing operations on the first composite image to obtain a second composite image; the second basic image is the first basic image; and obtaining third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0048] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: the second composite image is the first composite image; image processing operations are performed on the first basic image to obtain a second basic image; and third encoded data and fourth encoded data are obtained based on the second composite image and the second basic image.

[0049] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: perform image processing operations on the first basic image to obtain a second basic image; obtain a second composite image based on the second basic image; and obtain third encoded data and fourth encoded data based on the second composite image and the second basic image.

[0050] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: performing image processing operations on the first basic image to obtain a second basic image; performing image processing operations on the first enhanced image to obtain a second enhanced image; and obtaining third encoded data and fourth encoded data based on the second enhanced image and the second basic image.

[0051] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: encoding the second basic image to obtain third encoded data; encoding the second enhanced image to obtain fourth encoded data.

[0052] In one possible implementation, the one or more programs are executed by one or more processors, so that the one or more processors implement the following steps: obtaining an operating mode of an image processing operation, the operating mode including at least one of the following: enlarging, reducing, sharpening, blunting, adding a watermark, or removing a watermark.

[0053] In a possible implementation, the operation mode is preset, or the operation mode of the image processing operation is obtained, including: determining the operation mode based on a received user operation or user demand; or obtaining the operation mode from the first dual-layer code stream.

[0054] In one possible implementation, the difference includes at least one of the following: the resolution of the second composite image is different from that of the first composite image; or the clarity of the second composite image is different from that of the first composite image; or the image content of the second composite image has an additional watermark or an insufficient watermark compared to the first composite image.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium comprising a computer program, wherein when the computer program is executed on a computer, the computer is enabled to perform any of the methods described in the first aspect.

[0056] In a fifth aspect, the present application provides a computer program, which, when executed by a computer, is used to perform any of the methods described in the first aspect.

[0057] In a sixth aspect, the present application also provides a computer program product, which includes computer program code. When the computer program code runs on a computer, it enables the computer to execute the operations and / or processing performed by the electronic device in any of the above method embodiments.

[0058] In a seventh aspect, an embodiment of the present application provides a code stream structure, which includes encoded data of a first basic image, encoded data of a first enhanced image, and metadata.

[0059] In an eighth aspect, an embodiment of the present application provides a code stream structure, which includes encoded data of a first basic image, encoded data of a first enhanced image, and metadata, wherein the metadata includes tone mapping information. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] FIG1 is a schematic block diagram of a coding solution system used in an embodiment of the present application;

[0061] FIG2A is a block diagram illustrating an exemplary content providing system;

[0062] FIG2B is a schematic structural diagram of an exemplary terminal device;

[0063] FIG3A is a schematic diagram illustrating an exemplary workflow of a streaming media system;

[0064] FIG3B is a schematic diagram illustrating an exemplary streaming media system architecture;

[0065] FIG4A is an exemplary architecture diagram of an image processing system;

[0066] FIG4B is an exemplary architecture diagram of an image processing system;

[0067] FIG5 is a schematic diagram exemplarily illustrating dynamic mapping of a high dynamic range in the real world to a low dynamic range of a display device;

[0068] FIG6 is a schematic diagram illustrating an exemplary end-to-end system of a dual-layer code stream;

[0069] FIG7A is a system block diagram exemplarily illustrating image processing;

[0070] FIG7B is a block diagram of another system for image processing;

[0071] FIG8 is a flow chart showing an exemplary image processing method;

[0072] FIG9 is a schematic diagram illustrating an exemplary tone mapping process;

[0073] FIG. 10 is a schematic structural diagram illustrating an exemplary image processing apparatus. DETAILED DESCRIPTION

[0074] The term "at least one" in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where any of a, b, c, ab, ac, bc, or abc can include a single a, a single b, a single c, or multiple a, multiple b, or multiple c.

[0075] Furthermore, unless otherwise indicated, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish between multiple objects and are not used to define the order, timing, priority, or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria and do not indicate differences in the content, priority, or importance of the two criteria.

[0076] In addition, the terms "including" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules / units is not limited to the listed steps or modules and may also include steps or modules / units that are not listed.

[0077] First, a brief description of the background technology that may be involved in this application is given:

[0078] FIG1 is a schematic block diagram of a video encoding and decoding system used in embodiments of the present application. The video encoder 20 (or simply encoder 20 ) and video decoder 30 (or simply decoder 30 ) in the video encoding and decoding system 10 represent devices that can be used to perform various techniques according to the various examples described in this application.

[0079] As shown in FIG1 , a coding and decoding system 10 includes a source device 12 , which is configured to provide coded data 21 such as coded images to a destination device 14 for decoding the coded data.

[0080] The source device 12 includes an encoder 20 and, optionally, may further include an image source 16 , an image preprocessor 18 (or a preprocessing unit), and a communication interface or communication unit 22 .

[0081] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0082] In order to distinguish the processing performed by the pre-processor 18 or the pre-processing unit 18 , the image or image data 17 may also be referred to as a raw image or raw image data 17 .

[0083] The preprocessor 18 is configured to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.

[0084] The video encoder 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 .

[0085] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device (such as a transcoding end) through the communication channel 13 for storage or direct reconstruction.

[0086] Destination device 14 includes a decoder 30 (eg, video decoder 30 ) and, in addition and optionally, may include a communication interface or communication unit 28 , a post-processor 32 (or post-processing unit 32 ), and a display device 34 .

[0087] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, a storage device that is an encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0088] The communication interface 22 and the communication interface 28 may be used to send or receive the encoded image data 21 or the encoded data via a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any type of network, such as a wired network, a wireless network, or any combination thereof, any type of private network and public network, or any combination thereof.

[0089] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0090] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21 .

[0091] Both the communication interface 22 and the communication interface 28 can be configured as a unidirectional communication interface as indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 in Figure 1, or a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0092] The decoder 30 is configured to receive the encoded image data 21 and provide decoded image data 31 or a decoded image 31 .

[0093] The post-processor 32 in the destination device 14 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.

[0094] A display device 34 in the destination device 14 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0095] Although FIG1 shows source device 12 and destination device 14 as separate devices, device embodiments may also include both source device 12 and destination device 14 or the functionality of both source device 12 and destination device 14, that is, both source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality. In these embodiments, source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality may be implemented using the same hardware and / or software or through separate hardware and / or software, or any combination thereof.

[0096] Based on the description, it will be apparent to those skilled in the art that the existence and (precise) division of different units or functions in the source device 12 and / or the destination device 14 shown in FIG. 1 may vary depending on actual devices and applications.

[0097] FIG2A is a block diagram of an exemplary content provision system. The following describes the content provision system for the content distribution service used in this application in conjunction with FIG2A . The content provision system 20 includes a capture device 2102, a terminal device 2106, and an (optional) display 2126. The capture device 2102 communicates with the terminal device 2106 via a communication link 2104. The communication link may include the aforementioned communication channel 13. The communication link 2104 includes, but is not limited to, WiFi, Ethernet, wired, wireless (3G / 4G / 5G), USB, or any combination thereof.

[0098] Capture device 2102 generates data and encodes the data. Alternatively, capture device 2102 can distribute the data to a streaming media server (not shown), which encodes the data and transmits the encoded data to terminal device 2106. Capture device 2102 includes but is not limited to a camera, a smart phone or tablet computer, a computer or laptop computer, a video conferencing system, a PDA, a vehicle-mounted device, or any combination thereof. For example, capture device 2102 can include the above-mentioned source device 12. When the data includes video, the encoder 20 in capture device 2102 can actually perform video encoding processing. When the data includes audio (i.e., sound), the encoder 20 in capture device 2102 can actually perform audio encoding processing. In some actual scenarios, capture device 2102 distributes encoded video data and encoded audio data by multiplexing the encoded video data and the encoded audio data together. In other actual scenarios, such as in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. Capture device 2102 distributes the encoded audio data and the encoded video data to terminal device 2106 respectively.

[0099] The terminal device 2106 in the content providing system 20 receives and regenerates the encoded data. The terminal device 2106 can be a device with data reception and recovery capabilities, such as a smartphone or tablet 2108, a computer or laptop 2110, a network video recorder (NVR) / digital video recorder (DVR) 2112, a television 2114, a set-top box (STB) 2116, a video conferencing system 2118, a video surveillance system 2120, a personal digital assistant (PDA) 2122, an in-vehicle device 2124, or any combination thereof, or any other device capable of decoding the encoded data. For example, the terminal device 2106 can include the destination device 14 described above. When the encoded data includes video, the video decoder 30 in the terminal device prioritizes video decoding. When the encoded data includes audio, the audio decoder in the terminal device prioritizes audio decoding. The terminal device 2106 can be a video playback application, a streaming media playback application, a streaming media playback platform, or a live broadcast platform running on the terminal device.

[0100] For end devices with displays, such as smartphones or tablets 2108, computers or laptops 2110, NVRs / DVRs 2112, televisions 2114, PDAs 2122, or in-vehicle devices 2124, the end device can send the decoded data to its display. For end devices without displays, such as STBs 2116, video conferencing systems 3118, or video surveillance systems 2120, the device can be connected to an external display 2126 to receive and display the decoded data.

[0101] When performing encoding or decoding, each device in this system may use the image encoding device or image decoding device shown in the above embodiments.

[0102] FIG2B is a schematic diagram of an example structure of the terminal device 2106 in FIG2A . After the terminal device 2106 receives the bitstream from the capture device 2102, the protocol processing unit 2202 analyzes the transmission protocol of the bitstream. The protocol includes, but is not limited to, Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP), or any combination thereof.

[0103] After processing the stream, protocol processing unit 2202 generates a stream file. This file is output to demultiplexing unit 2204. Demultiplexing unit 2204 can separate the multiplexed data into coded audio data and coded video data. As mentioned above, in other practical scenarios, such as video conferencing systems, coded audio data and coded video data are not multiplexed. In this case, the coded data is not transmitted to video decoder 3206 and audio decoder 2208 via demultiplexing unit 2204.

[0104] Through the demultiplexing process, a video elementary stream (ES), an audio ES, and optional subtitles are generated. The video decoder 2206, including the video decoder 30 described in the above embodiment, decodes the video ES by the decoding method shown in the above embodiment to generate video frames, and sends the data to the synchronization unit 2212. The audio decoder 2208 decodes the audio ES to generate audio frames, and sends the data to the synchronization unit 2212. Alternatively, the video frames can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212. Similarly, the audio frames can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212.

[0105] Synchronization unit 2212 synchronizes video frames and audio frames and provides the video / audio to video / audio display 2214. For example, synchronization unit 2212 synchronizes the presentation of video and audio information. The information can be encoded in the syntax using timestamps associated with the representation of the encoded audio and visual data and timestamps associated with the transmission of the data stream.

[0106] If subtitles are included in the bitstream, the subtitle decoder 2210 decodes the subtitles, synchronizes them with the video frames and audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 2216.

[0107] The present invention is not limited to the above-mentioned system. The image encoding device or the image decoding device in the above-mentioned embodiments can be used in other systems such as automobiles.

[0108] FIG3A is a schematic diagram of a workflow of a streaming media system applicable to an embodiment of the present application. The streaming media system applicable to an embodiment of the present application is described below with reference to FIG3A.

[0109] The streaming media system includes a content creation module that generates the required content data, such as video or audio. The streaming media system also includes a video encoding module that encodes the generated content through an encoder. The streaming media system also includes a video stream transmission module that transmits the encoded video in the form of a code stream. Optionally, the video stream transcoding module can convert the format of the video stream into a code stream format of a commonly used transmission protocol for OTT (over-the-top) devices, for example, the protocol includes but is not limited to Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG HTTP Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP) or any combination thereof. Optionally, the video stream storage module can store the video stream and store the original format of the video stream and / or the converted multiple code stream formats for easy use. Furthermore, the streaming system also includes a video stream encapsulation module for encapsulating the video stream to generate an encapsulated video stream. The encapsulated video stream can be referred to as a video streaming package. Exemplarily, the video streaming package can be generated based on a transcoded video stream or a stored video stream. Furthermore, the streaming system also includes a content distribution network (CDN) for distributing the video streaming package to multiple OTT devices, such as mobile phones, computers, tablets, and home projectors.

[0110] It should be noted that video encoding, video streaming transmission, video stream transcoding, video stream storage, video streaming package generation and content distribution network can all be implemented on cloud servers.

[0111] FIG3B is a schematic diagram of an exemplary streaming media system architecture. An exemplary streaming media system architecture of the present application will be described below in conjunction with FIG3B . The streaming media system architecture includes: a client device, a content distribution network, and a cloud server.

[0112] The user on the client device sends a play or playback request to the cloud platform. Optionally, the content of the request can be the title of the movie or TV program to be played.

[0113] The cloud platform makes a decision and responds to the client, sending the client the address of the requested content on the CDN. Optionally, the content sent to the client can be a URL link (uniform resource locator). Specifically, the playback application service in the cloud platform checks user authorization and permissions, and then considers individual client characteristics and current network conditions to determine which specific files are needed to process the playback request. It should be noted that the content delivery network (CDN) regularly reports its operating status, learned routes, and available content (files) to the cache control service in the cloud platform.

[0114] The client then requests the CDN to play the content based on the address, and the CDN provides the content to the client, ultimately completing the client's request.

[0115] FIG4A is a schematic diagram of a possible system architecture applicable to an embodiment of the present application, wherein the system architecture of the embodiment of the present application includes: a front-end device, a transmission link, and a terminal display device. The system architecture applicable to the embodiment of the present application is described below with reference to FIG4A.

[0116] Among them, the front-end equipment is used to capture or produce HDR / SDR content (for example, HDR / SDR video or images).

[0117] In a possible embodiment, the front-end device can also be used to extract corresponding metadata from the HDR content, which may include global mapping information, local mapping information, and dynamic metadata and static metadata corresponding to the HDR content. The front-end device can send the HDR content and metadata to the terminal display device through a transmission link. Specifically, the HDR content and metadata can be transmitted in the form of a data packet (i.e., an HDR code stream), or respectively in two data packets (i.e., an HDR code stream and a metadata stream), which is not specifically limited in the embodiments of the present application.

[0118] Optionally, the terminal display device can be used to receive metadata and HDR content, and extract the global mapping information, local mapping information, and the terminal display device information contained in the corresponding metadata according to the HDR content, obtain a mapping curve to perform global tone mapping and local tone mapping on the HDR content, convert it into display content adapted for the HDR display device or SDR device in the terminal display device, and display it. It should be understood that in different embodiments, the terminal display device may include a display device with a display capability of a lower dynamic range or a higher dynamic range than the HDR content generated by the front-end device, and this application is not limited thereto.

[0119] Optionally, the front-end device and the terminal display device in this application can be independent and different physical devices. For example, the front-end device can be a video capture device or a video production device, where the video capture device can be a video camera, a camera, an image rendering machine, etc. The terminal display device can be a device with video playback function, such as virtual reality (VR) glasses, a mobile phone, a tablet, a television, a projector, etc.

[0120] Optionally, the transmission link between the front-end device and the terminal display device can be a wireless connection or a wired connection, wherein the wireless connection can adopt technologies such as long term evolution (LTE), fifth generation (5G) mobile communications, and future mobile communications. Wireless connections can also include wireless fidelity (WiFi), Bluetooth, near field communication (NFC), and other technologies. Wired connections can include Ethernet connections, local area network connections, etc. There is no specific limitation on this.

[0121] The present application may also integrate the functions of the front-end device and the terminal display device into the same physical device, for example, a terminal device such as a mobile phone or tablet with a video capture function. The present application may also integrate some functions of the front-end device and some functions of the terminal display device into the same physical device. This is not specifically limited.

[0122] The specific implementation of the end-to-end system in FIG4A can be referred to FIG1 , and will not be described again here.

[0123] FIG4B is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application. The end-to-end image processing system provided in an embodiment of the present application is described below in conjunction with FIG4B . This system can be applied to the system architecture shown in FIG4A . In FIG4B , HDR / SDR content is exemplified by HDR video. The image processing system includes: an HDR preprocessing module, an HDR video encoding module, an HDR video decoding module, and a tone mapping module. Optionally, the system may also include a transcoding module.

[0124] Among them, the HDR preprocessing module and the HDR video encoding module can be located in the front-end device shown in Figure 4A, and the HDR video decoding module and the tone mapping module can be located in the terminal display device shown in Figure 4A.

[0125] HDR pre-processing module: This module is used to extract dynamic metadata (e.g., maximum, minimum, average, and range of brightness) from HDR video, determine mapping curve parameters based on the dynamic metadata and the display capabilities of the target display device, write the mapping curve parameters into the dynamic metadata to obtain HDR metadata, and transmit the metadata. HDR video can be either captured or processed by a colorist; the display capabilities of the target display device are the brightness range that the target display device can display.

[0126] HDR video encoding module: used to encode HDR video and HDR metadata according to the video compression standard (for example, AVS or HEVC standard) (for example, embedding HDR metadata into the user-defined part of the bitstream) and output the corresponding bitstream (AVS or HEVC bitstream).

[0127] Transcoding module: used to receive the code stream. Transcode the code stream (Transcoding). Transcoding can optionally convert the compressed and encoded code stream into another code stream to adapt to different network bandwidths, different terminal processing capabilities, and / or different user needs. Transcoding can be understood as the process of first decoding the code stream and then re-encoding it. Optionally, the code streams before and after transcoding may follow the same video coding standard or different video coding standards. This application does not limit this and will not be repeated below.

[0128] HDR video decoding module: used to decode the generated bitstream (AVS bitstream or HEVC bitstream) according to the standard corresponding to the bitstream format, and output the decoded HDR video and HDR metadata.

[0129] Tone mapping module: used to generate a mapping curve according to the parameters of the mapping curve in the decoded HDR metadata, and perform tone mapping on the decoded HDR video (i.e. HDR adaptation processing or SDR adaptation processing), and send the HDR adapted video after tone mapping to the HDR display terminal for display, or send the SDR adapted video to the SDR display terminal for display.

[0130] Exemplarily, the HDR pre-processing module may exist in a video acquisition device or a video production device.

[0131] Exemplarily, the HDR video encoding module may exist in a video acquisition device or a video production device.

[0132] Exemplarily, the HDR video decoding module may be present in a set-top box, a television display device, a mobile terminal display device, and a video conversion device for live broadcasting, online video applications, and the like.

[0133] For example, the tone mapping module may be present in a set-top box, a television display device, a mobile terminal display device, or a video conversion device for a live webcast or online video application. More specifically, the tone mapping module may be present in the form of a chip or software program in the set-top box, television display, or mobile terminal display, and may be present in the form of a software program in the video conversion device for a live webcast or online video application.

[0134] In a possible embodiment, when the tone mapping module and the HDR video decoding module are both present in a set-top box, the set-top box can complete the functions of receiving, decoding, and tone mapping the video stream. The set-top box sends the decoded video data to a display device through a high-definition multimedia interface (HDMI) for display, so that the user can enjoy the video content.

[0135] The following is an explanation of the technical terms that may be involved in this application:

[0136] 1) High Dynamic Range (HDR)

[0137] Dynamic range refers to the ratio of maximum to minimum brightness in a video or image signal. In many fields, dynamic range is used to express the ratio of the maximum to minimum values ​​of a variable. In digital imaging, dynamic range represents the ratio between the maximum and minimum grayscale values ​​within the image's displayable range. The dynamic range in nature is quite large. The brightness of a starry night sky is approximately 0.001 cd / m², while the sun itself has a brightness of 1,000,000,000 cd / m², resulting in a dynamic range of 1,000,000,000 / 0.001 = 1013. However, in real-world scenes, the brightness of the sun and starlight are not present at the same time. For real-world scenes, the dynamic range is between 10-3 and 106. In most current color digital images, each R, G, and B channel is stored using an 8-bit byte. This means each channel represents a grayscale range of 0 to 255. This range, 0 to 255, represents the image's dynamic range. Since the dynamic range of a scene in the real world ranges from 10⁻³ to 10⁶, this is called High Dynamic Range (HDR). The dynamic range of an ordinary image is referred to as Low Dynamic Range (LDR). The imaging process of a digital camera is essentially a mapping from the real-world high dynamic range to the photographic low dynamic range. This mapping is often a nonlinear process.

[0138] Figure 5 shows a schematic diagram of a dynamic mapping of a high dynamic range in the real world to a low dynamic range of a display device. In Figure 5 , the dynamic range of the real world is about 80 to 2000, and the dynamic range mapped to the display device is about 1 to 200.

[0139] 2) Standard dynamic range (SDR) image and its photoelectric conversion function

[0140] Standard dynamic range (SDR) images, also known as low dynamic range, typically have a dynamic range between 1 nit and 100 nits. Images with a dynamic range between 1 nit and 100 nits conform to the BT.709 or sRGB color gamut and use a gamma curve as the photoelectric conversion curve.

[0141] Standard dynamic range images correspond to high dynamic range images. Traditionally used 8-bit images in formats such as JPEG can be considered standard dynamic range images. Before the advent of cameras capable of capturing HDR images, traditional cameras could only record captured light information within a certain range by controlling the exposure value. Because the maximum brightness information of a display device cannot match the brightness information in the real world, and we use display devices to browse images, a photoelectric transfer function is required. Early display devices were cathode ray tube (CRT) displays, whose photoelectric transfer function was the gamma function. This photoelectric transfer function, based on the "gamma" function, is defined in the ITU-R Recommendation BT.1886 standard.

[0142] 3) HDR photoelectric conversion function

[0143] With the upgrade of display devices, the illumination range of display devices continues to increase. The existing consumer-grade HDR display is 600cd / m2, and the illumination information of high-end HDR display can reach 2000cd / m2, which far exceeds the illumination information of SDR display devices. The photoelectric conversion function in the ITU-R Recommendation BT.1886 standard cannot well express the display performance of HDR display devices. Therefore, an improved photoelectric transfer function is needed to adapt to the upgrade of display devices. At present, there are three common photoelectric conversion functions: Perception quantization (PQ), Hybrid Log-Gamma (HLG) and Scene Luminance Fidelity (SLF) photoelectric conversion functions. The above three curves can refer to the existing technology and will not be described in detail in this application.

[0144] 4) HDR end-to-end process

[0145] The dynamic range mapping method mainly adapts the front-end HDR signal and the back-end HDR terminal display device through dynamic range mapping. For example, the front-end collects a 4000nit light signal (nit is the unit of light intensity), while the back-end HDR terminal display device (TV, iPad) has an HDR display capability of only 500nit. Therefore, how to map the 4000nit signal to a 500nit device is a high-to-low tone-mapping (Tone Mapping, TM) process. Another example is that the front-end collects a 100nit SDR signal, while the display end is a 2000nit TV signal. If the 100nit signal is better displayed on a 2000nit device, it is another low-to-high tone-mapping process.

[0146] Dynamic range mapping can be divided into static and dynamic. The static mapping method is to perform the overall tone-mapping process based on a single piece of data based on the same video content or the same hard disk content, that is, the processing curve is usually the same. The advantage of this method is that it carries less information and the processing flow is relatively simple; the disadvantage of this method is that each scene uses the same curve for tone-mapping, which will lead to information loss in some scenes. For example, if the curve focuses on protecting bright areas, some details will be lost in some extremely dark scenes, or simply cannot be seen, which will affect the experience. The dynamic mapping method is to dynamically adjust according to the content of a specific area, each scene, or each frame. The advantage of this method is that different curve processing can be performed according to a specific area, each scene, or each frame, so that the processing results will be better. The disadvantage is that each frame or each scene needs to carry relevant scene information, and the amount of information carried is relatively large.

[0147] 5) A dual-layer code stream, including basic image coding data (for example, the first coding data and the third coding data in the embodiment of the present application) and enhanced image coding data (for example, the second coding data and the fourth coding data in the embodiment of the present application). Among them, the enhanced image coding data is used to represent the enhanced image (which may also be referred to as the enhanced layer image, etc., which is not limited in this application), and the basic image coding data is used to represent the basic image (for example, which may also be referred to as the basic layer image, the basic image, the base layer image, etc., which is not limited in this application). In the embodiment of the present application, the dual-layer code stream may include one or more enhanced images, and one or more enhanced images. In the following embodiments, in the process of processing images (including enhanced image data or basic image data), unless otherwise specified, it means that the same processing is performed on each image data. In the embodiment of the present application, the dual-layer code stream can be an image (or picture) code stream or a video code stream. That is, the enhanced image and the basic image in the dual-layer code stream can be pictures or videos, that is, the image processing method in the embodiment of the present application can be applied to the processing of image code streams or to the processing of video code streams. Optionally, in some examples, the dual-layer code stream can also be understood as encapsulating the enhanced image code stream and the basic image code stream into one code stream, that is, after the receiving end or transcoding end receives the dual-layer code stream and processes the dual-layer code stream (which can be called decapsulation processing), the basic image code stream and the enhanced image code stream can be obtained. In other words, the various processing of the enhanced image described in the embodiments of the present application (including decoding, image processing and encoding, etc.) can be understood as processing the enhanced image stream (i.e., one or more enhanced images). The various processing of the basic image (including decoding, image processing and encoding, etc.) can be understood as processing the basic image stream (i.e., one or more basic images). This application does not limit this and will not be repeated below.

[0148] 6) Single-layer code stream, which can be an enhanced image code stream or a basic image code stream. Among them, the enhanced image code stream includes one or more enhanced images. The basic image code stream includes one or more basic images. Optionally, the HDR video (or HDR image) in the existing technology can be encoded and transmitted using H.265, etc., and the single-layer video (i.e., single-layer code stream) is transcoded during the transmission process. After decoding, the HDR video obtains frame-by-frame HDR pixel values ​​and HDR format identification and information. HDR video usually needs to be sent to the image processing module to complete image upsampling, downsampling, enhancement and other operations. After processing, the HDR pixel value and HDR format identification and information are sent to the encoder for re-encoding.

[0149] 7) Base image: describes an independent image data structure, including pixel and image-related metadata.

[0150] 8) Enhancement image: describes the enhanced image data structure, including pixel and image-related metadata.

[0151] 9) Synthetic image, which may also be referred to as a derived alternate image, selectable image, replacement image, or substitute image in the Ultra High Definition World Association (UWA) standard, and is not limited in this application. Synthetic image is used to describe the image data structure obtained from the format specified in the embodiments of this application for display or subsequent processing, including pixel and image-related metadata.

[0152] 10) Metadata: data that describes image attributes and characteristics, as well as key information required during image processing.

[0153] Figure 6 is a schematic diagram illustrating an exemplary end-to-end system for a dual-layer stream. As shown in Figure 6, a dual-layer stream end-to-end processing system (also referred to as a dual-layer distribution format end-to-end system, not limited in this application) includes, but is not limited to: a transmitter (also referred to as an encoder, generator, etc., not limited in this application), a transcoder, and a receiver (also referred to as a decoder or display, not limited in this application).

[0154] On the generation side, the enhanced image generation module obtains DRE-AIG (dynamic range extending associated image group) data (including the base image, enhanced image, and metadata) from the base image data and the synthesized image data. Next, the encoding module generates a base image stream, an enhanced image stream, and a metadata stream. Then, a dual-layer stream (which can be understood as the first dual-layer stream described in the embodiment of the present application) is generated for the encapsulated image and metadata streams, which can also be called an HDR static image dual-layer distribution format file.

[0155] In the embodiment of the present application, the generating end transmits the first double-layer code stream to the transcoding end (also referred to as the distributing end).

[0156] At the transcoding end, a first dual-layer stream is received. A transcoding operation is performed on the first dual-layer stream, and a transcoded dual-layer stream is output (which can be understood as the second dual-layer stream described in the embodiments of this application). The specific operation process will be described in detail in the embodiments below. The first dual-layer stream and the second dual-layer stream can follow the same or different encoding formats, which is not limited in this application.

[0157] After receiving the second dual-layer stream, the receiver decapsulates it, extracting the base image stream, the enhanced image stream, and the metadata stream. The base image stream and the enhanced image stream are then decoded to obtain the DRE-AIG data, the enhanced image data, and the metadata. The resulting composite image is then processed for display adaptation and displayed. (For the display end, see the previous section and will not be further elaborated here.)

[0158] Optionally, in some instances, the transcoding end processes the code stream and saves or transmits it to the decoding end. Optionally, the encoding end and the transcoding end can be set on different devices or on the same device (for example, both are set on the sending end), which is not limited in this application.

[0159] FIG7A is a system block diagram of an exemplary image processing. An embodiment of the present application provides an image processing method applicable to a transcoding end. As described above, transcoding may optionally convert a compressed and encoded code stream into another code stream to adapt to different network bandwidths, different terminal processing capabilities, and / or different user needs. In the process of image processing on the image data, the transcoding end in the embodiment of the present application may perform corresponding image processing on the double-layer code stream according to the requirements of different scenarios, and may meet the requirements for the transcoded double-layer code stream in different scenarios. For example, in some scenarios, the third-party platform to which the transcoding end belongs may perform downsampling and other processing on the image in consideration of storage costs to reduce the resolution of the image, thereby reducing the storage space occupied by the image data and reducing the bandwidth occupied by the transmission. For another example, in some scenarios, the platform has high requirements for the quality of the image data, and the resolution of the image data may be increased during the transcoding process to improve the quality of the transcoded image data.

[0160] In the embodiments of the present application, the image processing method corresponds to the scene requirements. The scene requirements may be set by the user, the platform (i.e., pre-set by the transcoding end, or set by the platform operator), or the sending end, and this application does not limit them. Detailed description can be found below.

[0161] Please refer to Figure 7A, the transcoding end (which may also be called a transcoding device, a transcoding equipment, a transcoding module, a transcoding server, etc., and is not limited in this application) may optionally include but is not limited to: a decapsulation module, a decoding module, an image processing module, an encoding module and an encapsulation module, etc.

[0162] At the transcoding end, a first code stream (also referred to as a first dual-layer code stream, not limited in this application) is received. The first code stream includes, but is not limited to, first base image coded data, first enhanced image coded data, and first metadata coded data. Optionally, the metadata coded data may also be transmitted via a separate metadata code stream, not limited in this application.

[0163] The decapsulation module (which can also be considered as one module with the encapsulation module) is used to decapsulate the first code stream and output first basic image coded data, first enhanced image coded data, and first metadata coded data.

[0164] The decoding module (which may also be referred to as an image decoding module, not limited in this application) is configured to decode the first basic image coded data to obtain a first basic image, decode the first enhanced image coded data to obtain a first enhanced image, and decode the first metadata coded data to obtain first metadata.

[0165] In this embodiment of the present application, each codestream includes at least one image (also referred to as image data). For example, decoding an enhanced image codestream yields at least one enhanced image (enhance1 ... enhanceN), and decoding a base image codestream yields at least one base image (base1 ... baseN). In this embodiment of the present application, the term "image data" is used to refer to the collection of at least one enhanced image and at least one base image, and this description will not be repeated below.

[0166] Optionally, the decoding module may decode the code stream using a decoder such as High Efficiency Video Coding (HEVC) or Joint Photographic Experts Group JPEG.

[0167] In one possible implementation, the image data (including the enhanced image and the base image) acquired by the decoding module can be image data in any color space format, such as RGB or YUV, and this application does not limit this. The color space format of the enhanced image and the base image can be the same or different, and this application does not limit this.

[0168] In another possible implementation, the bit width of the image data may be 8 bits, 10 bits, 12 bits, etc., which is not limited in this application.

[0169] In an embodiment of the present application, metadata can be embedded into a user-defined portion of the code stream during encoding. For example, the customized portion can be a Supplemental Enhancement Information (SEI) field in HEVC or Versatile Video Coding (VVC), or a customized Network Abstract Layer (NAL) unit, or other reserved fields. The customized portion can also be a JEIF-encapsulated APP extended information field or an MP4-encapsulated data segment, etc., which can be set according to actual needs and is not limited by this application.

[0170] In the embodiment of the present application, the metadata includes but is not limited to at least one of the following: source data format, region division information, region traversal order information, image features, tone mapping information and other data.

[0171] In the embodiments of the present application, the tone mapping information includes, but is not limited to, at least one of the following: tone mapping parameters, target image processing methods, and adjustment parameters. The tone mapping parameters include, but are not limited to, at least one of the following: base image tone mapping parameters and enhanced image tone mapping parameters. The specific usage will be described in detail below.

[0172] 7A , the image processing module is configured to perform image processing operations on the first basic image and the first enhanced image, and output a second basic image and a second enhanced image after image processing.

[0173] An encoding module is used to encode the second basic image, the second enhanced image, and the first metadata to obtain an encoded basic image (for example, recorded as second basic image encoding data, or third encoding data), an enhanced image (for example, recorded as second enhanced image encoding data, or fourth encoding data), and metadata (for example, recorded as second metadata encoding data).

[0174] The encapsulation module is used to encapsulate the encoded basic image, enhanced image and metadata and output a second code stream. The second code stream includes the second basic image encoded data, the second enhanced image encoded data and the second metadata encoded data.

[0175] Figure 7B is a system block diagram of another image processing system, in which the transcoding end (which may also be referred to as a transcoding device, a transcoding equipment, a transcoding module, a transcoding server, etc., which is not limited in this application) may optionally include but is not limited to: a decapsulation module, a decoding module, an enhanced image generation module, an image processing module, an encoding module and an encapsulation module, etc.

[0176] At the transcoding end, a first code stream (also referred to as a first dual-layer code stream, not limited in this application) is received. The first code stream includes, but is not limited to, first base image coded data, first enhanced image coded data, and first metadata coded data. Optionally, the metadata may also be transmitted via a separate metadata code stream, not limited in this application.

[0177] The decapsulation module (which can also be considered as one module with the encapsulation module) is used to decapsulate the first code stream and output first basic image coded data, first enhanced image coded data, and metadata coded data.

[0178] The decoding module (also referred to as the image decoding module, not limited in this application) is configured to decode the first basic image coded data to obtain a first basic image, decode the first enhanced image coded data to obtain a first enhanced image, and decode the metadata coded data to obtain metadata.

[0179] In this embodiment of the present application, each codestream includes at least one image (also referred to as image data). For example, after decoding the enhanced image coded data, at least one enhanced image (enhance1 ... enhanceN) can be obtained, and after decoding the base image coded data, at least one base image (base1 ... baseN) can be obtained. In this embodiment of the present application, the term "image data" is used to refer to the combination of at least one enhanced image and at least one base image, and this description will not be repeated below.

[0180] Optionally, the decoding module may decode the code stream using a decoder such as High Efficiency Video Coding (HEVC) or Joint Photographic Experts Group JPEG.

[0181] In one possible implementation, the image data (including the enhanced image and the base image) acquired by the decoding module can be image data in any color space format, such as RGB or YUV, and this application does not limit this. The color space format of the enhanced image and the base image can be the same or different, and this application does not limit this.

[0182] In another possible implementation, the bit width of the image data may be 8 bits, 10 bits, 12 bits, etc., which is not limited in this application.

[0183] In an embodiment of the present application, metadata can be embedded into a user-defined portion of the code stream during encoding. For example, the customized portion can be a Supplemental Enhancement Information (SEI) field in HEVC or Versatile Video Coding (VVC), or a customized Network Abstract Layer (NAL) unit, or other reserved fields. The customized portion can also be a JEIF-encapsulated APP extended information field or an MP4-encapsulated data segment, etc., which can be set according to actual needs and is not limited by this application.

[0184] In the embodiment of the present application, the metadata includes but is not limited to at least one of the following: source data format, region division information, region traversal order information, image features, tone mapping information and other data.

[0185] In the embodiments of the present application, the tone mapping information includes, but is not limited to, at least one of the following: tone mapping parameters, target image processing methods, and adjustment parameters. The tone mapping parameters include, but are not limited to, at least one of the following: base image tone mapping parameters and enhanced image tone mapping parameters. The specific usage will be described in detail below.

[0186] Continuing with FIG7B , the enhanced image generation module is configured to obtain a composite image (referred to as the first composite image in this embodiment of the present application) based on the base image (i.e., the first base image) and the enhanced image (i.e., the first enhanced image). The enhanced image generation module inputs the base image and the composite image into the image processing module.

[0187] The image processing module is configured to perform image processing on image data and output the processed image data. The image data includes at least one of a base image, an enhanced image, and a composite image. In this example, the base image and the composite image serve as input to the image processing module. During image processing by the image processing module, the image processing module may perform image processing on the base image and / or the composite image.

[0188] Optionally, the image processing module performs image processing on the basic image (for example, in the embodiment of the present application, it can be recorded as the first basic image) and the composite image (for example, in the embodiment of the present application, it can be recorded as the first composite image), and outputs the processed composite image (for example, in the embodiment of the present application, it can be recorded as the second composite image) and the basic image (for example, in the embodiment of the present application, it can be recorded as the second basic image).

[0189] Optionally, the image processing module may determine a target processing method and perform image processing on the image data according to the target processing method.

[0190] The enhanced image generation module is used to obtain an enhanced image (which may be referred to as the second enhanced image in the embodiment of the present application) based on the basic image (ie, the second basic image) and the synthesized image (ie, the second synthesized image).

[0191] The encoding module is used to encode the basic image (i.e., the second basic image), the enhanced image (i.e., the second enhanced image), and the metadata, and output the encoded basic image (for example, in the embodiment of the present application, recorded as the second basic image encoding data, or the third encoding data, used to represent the second basic image), the enhanced image (for example, in the embodiment of the present application, recorded as the second enhanced image encoding data, or the fourth encoding data, used to represent the second enhanced image), and the metadata.

[0192] The encapsulation module is used to output a second code stream (also called a second dual-layer code stream, which is not limited in this application) based on the encoded basic image, enhanced image and metadata.

[0193] Optionally, the forwarding end may further include a communication module (or may be called a transceiver module) configured to receive the first code stream or send the second code stream.

[0194] Optionally, the forwarding section may further include a storage module for storing the second code stream.

[0195] Exemplarily, the receiving end receives the second code stream and performs operations such as decapsulation and decoding on the second code stream to obtain a second basic image and a second enhanced image. The receiving end may generate a second composite image based on the second basic image and the second enhanced image. Specifically, the receiving end may merge the second basic image and the second enhanced image according to the metadata to obtain the second composite image. The second composite image generated by the receiving end is the same as the second composite image output by the image processing module in Figure 7B. It can be understood that the second basic image and the second enhanced image generated by the transcoding end can be used to synthesize the second composite image. Optionally, in some instances, the transcoding end may directly process the first basic image and the first enhanced image during the image processing process to obtain the second basic image and the second enhanced image. That is, in this example, the transcoding end does not generate the first composite image and the second composite image. In this example, the first basic image and the first enhanced image can still be understood as being used to synthesize the first composite image. Similarly, the second basic image and the second enhanced image are used to synthesize the second composite image.

[0196] FIG8 is a flowchart of an exemplary image processing method. Referring to FIG8 , the method specifically includes but is not limited to the following steps:

[0197] S801, obtaining a first double-layer code stream, the first double-layer code stream including first coded data and second coded data, the first coded data being used to represent a first basic image, the second coded data being used to represent a first enhanced image, the first basic image and the first enhanced image being used to synthesize a first synthesized image.

[0198] Exemplarily, the transcoding end obtains a first code stream. In an embodiment of the present application, the first code stream is a dual-layer code stream, including encoded data of a base image (i.e., first encoded data) and encoded data of an enhanced image (i.e., second encoded data). To distinguish the base image from subsequent base images and enhanced images, the base image in the first code stream is referred to as a first base image, and the enhanced image is referred to as a first enhanced image. Optionally, the first base image may include one or more base images. The first enhanced image may include one or more enhanced images.

[0199] Optionally, the first code stream includes metadata encoding data. In other embodiments, the metadata may also be a separate code stream, which is not limited in this application.

[0200] Exemplarily, the decapsulation module decapsulates the first bitstream and outputs first enhanced image coded data, first base image coded data, and metadata coded data. Next, the decoding module decodes each bitstream according to the standard corresponding to the bitstream format and outputs image data (including the first enhanced image and the first base image) and metadata.

[0201] The decoding module decodes the encoded data to obtain a first basic image and a first enhanced image.

[0202] In the embodiment of the present application, the first basic image and the first enhanced image can be used to synthesize a first synthesized image (which can be recorded as HDRPic). The specific synthesis method will be described in detail below.

[0203] S802, performing an image processing operation based on the first double-layer code stream to obtain third coded data and fourth coded data, where the third coded data is used to represent the second basic image, and the fourth coded data is used to represent the second enhanced image. The second basic image and the second enhanced image are used to synthesize a second composite image, and there are differences associated with the image processing operation between the second composite image and the first composite image.

[0204] Exemplarily, the enhanced image generation module performs tone mapping processing on at least one first basic image and at least one first enhanced image based on the tone mapping parameters in the metadata, and then merges the at least one first basic image and the at least one first enhanced image after the tone mapping processing to obtain a first composite image.

[0205] The tone mapping processing of the first basic image and the first enhanced image will be described in detail below.

[0206] 1) Perform tone mapping on the basic image according to the basic image tone mapping parameters to obtain a tone mapped basic image (denoted as baseAfter).

[0207] Figure 9 is a schematic diagram illustrating an exemplary tone mapping process. Referring to Figure 9 , the enhanced image generation module performs tone mapping on at least one base image (base_1…base_N) based on the base image tone mapping parameters, generating at least one tone-mapped base image (denoted as baseAfter_1…baseAfter_N). It should be noted that this step only uses the processing of one base image (e.g., base_1) as an example. The same steps are used for processing other base images, and will not be described individually in this application.

[0208] Exemplary tone mapping processing methods for the basic image include but are not limited to the following methods:

[0209] Method 1:

[0210] baseAfter[i]=Base[i], where i is a pixel in the image.

[0211] In this method, the basic image is the tone-mapped image, which can also be understood as no tone mapping processing is performed.

[0212] Method 2:

[0213] baseAfter[i]=base[i]*THH+(A-base[i])*THL

[0214] Here, THH is the upper limit of the base image, and THL is the lower limit of the base image. It can be understood that THH and THL are used to indicate the numerical range of the base image. A is the maximum value stored in the base image. It can be understood that method 2 normalizes the base image to the range indicated by THH and THL. When the base image base is normalized to between 0 and 1.0, the value of A is 1.0.

[0215] Optionally, THH and THL are tone mapping parameters, i.e., included in the tone mapping information in the metadata. Optionally, A is a value pre-stored by the decoding module. That is, the encoder performs encoding based on this A value, and the transcoder also performs tone mapping based on this A value. The specific value can be set based on actual needs and is not limited by this application.

[0216] In some examples, THH and THL may also be pre-stored in the decoding module, that is, the agreed values ​​between the transcoding end and the encoding end. They can be set according to actual needs and are not limited in this application.

[0217] Method 3:

[0218] baseAfter[i]=base[i]+THL

[0219] Here, THL is the lower limit of the basic image. It can be understood that THL is used to indicate the numerical range of the basic image, that is, the range that is not less than THL. It can be understood that method three is to normalize the basic image so that its value is normalized to be within the numerical range not less than THL.

[0220] Optionally, THL is a tone mapping parameter, that is, included in the tone mapping information of the metadata. In some examples, THL can also be pre-stored in the enhanced image generation module, that is, a value agreed upon between the transcoding end and the encoding end.

[0221] Method 4:

[0222] baseAfter[i]=THH+A-base[i]

[0223] THH is the upper limit of the base image. It can be understood that THH indicates the range of values ​​for the base image, i.e., the range that does not exceed (or exceeds) THH. A is the maximum value stored for the base image. It can be understood that method 2 normalizes the base image to a value that does not exceed the range indicated by THH. When the base image base is normalized to between 0 and 1.0, the value of A is 1.0.

[0224] Optionally, THH is a tone mapping parameter, that is, included in the tone mapping information of the metadata. In some examples, THH can also be pre-stored in the enhanced image generation module, that is, a value agreed upon between the transcoding end and the encoding end.

[0225] Method 5:

[0226] baseAfter[i]=TMB(base[i])

[0227] Wherein, TMB() is a tone mapping parameter. Optionally, TMB() can be a global or local tone mapping function. In this manner, the decoding module can perform global tone mapping processing or local tone mapping processing on the base based on the TMB() function.

[0228] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TMB() function for expressing a mapping relationship, and the decoding function may determine the corresponding TMB() function based on these parameters.

[0229] In some examples, TMB() may also be pre-stored in the decoding module, that is, the agreed value between the transcoding end and the encoding end.

[0230] Method 6:

[0231] baseAfter[i]=TMB()*base[i]

[0232] Wherein, TMB() is a tone mapping parameter. Optionally, TMB() can be a global or local tone mapping function. In this manner, the decoding module can perform global tone mapping processing or local tone mapping processing on the base based on the TMB() function.

[0233] In this way, tone mapping processing based on the tone mapping function can make the subsequent synthesized image obtained based on baseAfter closer to the original synthesized image (i.e., the synthesized image generated by the transmitter), thereby reducing the data volume of the enhanced image in the second code stream (the concept of which can be seen below).

[0234] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TMB() function for expressing a mapping relationship, and the decoding function may determine the corresponding TMB() function based on these parameters.

[0235] In some examples, TMB() may also be pre-stored in the enhanced image generation module, that is, a value agreed upon between the transcoding end and the encoding end.

[0236] Method 7:

[0237] baseAfter[i]=TMB(TMB1(base[i]))

[0238] Among them, TMB() and TMB1() are both tone mapping parameters. Optionally, TMB() can be a global and or local tone mapping function. Optionally, TMB1() can be in the form of linear, spline, segmented curve, etc. The role of TMB1() can be understood as being used to indicate that the basic image base[i] is transformed into a specified range. That is to say, the tone mapping parameters in Methods 2 to 4 (such as THL or THH) can also be expressed using TMB1(), which can be understood as the decoding module being able to obtain the corresponding TMB1() based on the tone mapping parameters (such as THH, etc.). Of course, TMB1() in Method 7 (and Method 8) can also be expressed in any of Methods 2 to 4, and this application does not limit it.

[0239] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TMB() and TMB1() functions for expressing mapping relationships, and the decoding function may determine the corresponding TMB() and TMB1() functions based on these parameters.

[0240] In some examples, TMB() and / or TMB1() may also be pre-stored in the enhanced image generation module, that is, the agreed values ​​between the transcoding end and the encoding end.

[0241] Method 8:

[0242] baseAfter[i]=TMB()*TMB1(base[i])

[0243] Among them, TMB() and TMB1() are both tone mapping parameters. Optionally, TMB() can be a global and or local tone mapping function. Optionally, TMB1() can be in the form of linear, spline, segmented curve, etc. The role of TMB1() can be understood as being used to indicate that the basic image base[i] is transformed into a specified range. That is to say, the tone mapping parameters in Methods 2 to 4 (such as THL or THH) can also be expressed using TMB1(), which can be understood as the decoding module being able to obtain the corresponding TMB1() based on the tone mapping parameters (such as THH, etc.). Of course, TMB1() in Method 7 (and Method 8) can also be expressed in any of Methods 2 to 4, and this application does not limit it.

[0244] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TMB() and TMB1() functions for expressing mapping relationships, and the decoding function may determine the corresponding TMB() and TMB1() functions based on these parameters.

[0245] In some examples, TMB() and / or TMB1() may also be pre-stored in the enhanced image generation module, that is, the agreed values ​​between the transcoding end and the encoding end.

[0246] Method 9:

[0247] baseAfter[i]=TMB()*TMB1(baseMid1[i]);

[0248] or,

[0249] baseAfter[i]=TMB(TMB1(baseMid1[i]));

[0250] in,

[0251] baseMid1[i]=F[base[i]] or ∑F[n]*base[i+n], baseAfter2[i]=base[i]-baseMid1[i]

[0252] Among them, the tone mapping parameters include but are not limited to: TMB(), TMB1() and F[]. Optionally, TMB() can be a global and or local tone mapping function. Optionally, TMB1() can be in the form of linear, spline, segmented curve, etc. The role of TMB1() can be understood as being used to indicate that the basic image base[i] is transformed into a specified range. That is to say, the tone mapping parameters in Methods 2 to 4 (such as THL or THH) can also be expressed using TMB1(), which can be understood as the decoding module being able to obtain the corresponding TMB1() based on the tone mapping parameters (such as THH, etc.). Of course, TMB1() in Method 7 (and Method 8) can also be expressed in any of Methods 2 to 4, and this application does not limit it.

[0253] Optionally, F[] is a spatial filter parameter or other image smoothing function. The enhanced image generation module can obtain baseMid1[i] and baseAfter2[i] based on F[]. There are various filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not limited in this application.

[0254] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TMB(), TMB1() and F[] functions for expressing mapping relationships, and the decoding function may determine the corresponding TMB(), TMB1() and F[] functions based on these parameters.

[0255] In some examples, at least one of TMB(), TMB1(), and F[] may also be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0256] Method 10:

[0257] baseAfter[i]=F[base[i]] or ∑F[n]*base[i+n], baseAfter2[i]=base[i]-baseMid1[i]

[0258] Where F[] is a spatial filter parameter or other image smoothing function. The enhanced image generation module can obtain baseMid1[i] and baseAfter2[i] based on F[]. Various filtering methods exist, such as bilateral filtering and interpolation filtering, which are not limited in this application.

[0259] Optionally, the tone mapping parameters of the metadata may also include some parameters in the F[] function for expressing the mapping relationship, and the decoding function may determine the corresponding F[] function based on these parameters.

[0260] In some examples, F[] may be pre-stored in the enhanced image generation module, that is, an agreed value between the transcoding end and the encoding end.

[0261] 2) Performing tone mapping processing on the enhanced image according to the enhanced image tone mapping parameters to obtain a tone mapped enhanced image (denoted as enhanceAfter).

[0262] Specifically, still referring to FIG9 , the enhanced image generation module performs tone mapping on at least one first enhanced image (enhance_1 ... enhance_N) based on the enhanced image tone mapping parameters, thereby obtaining at least one tone-mapped enhanced image (denoted as enhanceAfter_1 ... enhanceAfter_N). It should be noted that this step only uses the processing of one enhanced image (e.g., enhance_1) as an example for description. The same steps are used for processing other enhanced images, and will not be described individually in this application.

[0263] Exemplary tone mapping processing methods for enhancing an image include, but are not limited to, the following methods:

[0264] Method 1:

[0265] enchenceAfter[i]=enchence[i], where i is a pixel in the image.

[0266] In this method, the enhanced image is a tone-mapped image, which can also be understood as not performing tone mapping processing.

[0267] Method 2:

[0268] enhanceAfter[i]=enhance[i]*THH+(A-enhance[i])*THL

[0269] THH is the upper limit of the enhanced image, and THL is the lower limit. It can be understood that THH and THL indicate the numerical range of the enhanced image. A is the maximum value stored for the enhanced image. It can be understood that method 2 normalizes the enhanced image to the range indicated by THH and THL. When the enhanced image "enhance" is normalized to between 0 and 1.0, the value of A is 1.0.

[0270] Optionally, THH and THL are tone mapping parameters, i.e., included in the tone mapping information in the metadata. Optionally, A is a value pre-stored by the decoding module. That is, the encoder performs encoding based on this A value, and the transcoder also performs tone mapping based on this A value. The specific value can be set based on actual needs and is not limited by this application.

[0271] In some examples, THH and THL may also be pre-stored in the decoding module, that is, the agreed values ​​between the transcoding end and the encoding end. They can be set according to actual needs and are not limited in this application.

[0272] Method 3:

[0273] enhanceAfter[i]=enhance[i]+THL

[0274] Where THL is the lower limit of the enhanced image. It can be understood that THL is used to indicate the numerical range of the enhanced image, that is, the range that is not less than THL. It can be understood that method three is to normalize the enhanced image so that its value is normalized to be within the numerical range not less than THL.

[0275] Optionally, THL is a tone mapping parameter, that is, included in the tone mapping information of the metadata. In some examples, THL can also be pre-stored in the enhanced image generation module, that is, a value agreed upon between the transcoding end and the encoding end.

[0276] Method 4:

[0277] enhanceAfter[i]=THH+A-enhance[i]

[0278] THH is the upper limit of the enhanced image. It can be understood that THH indicates the numerical range of the enhanced image, that is, the range that does not exceed (or exceeds) THH. A is the maximum value stored for the enhanced image. It can be understood that method 2 normalizes the enhanced image to a value not exceeding the range indicated by THH. When the enhanced image enhance is normalized to between 0 and 1.0, the value of A is 1.0.

[0279] Optionally, THH is a tone mapping parameter, that is, included in the tone mapping information of the metadata. In some examples, THH can also be pre-stored in the enhanced image generation module, that is, a value agreed upon between the transcoding end and the encoding end.

[0280] Method 5:

[0281] enhanceAfter[i]=TME(enhance[i])

[0282] Wherein, TME() is a tone mapping parameter. Optionally, TME() can be a global or local tone mapping function. In this manner, the decoding module can perform global tone mapping or local tone mapping on the enhanced image based on the TME() function.

[0283] Optionally, the tone mapping parameters of the metadata may also include some parameters in the TME() function for expressing a mapping relationship, and the decoding function may determine the corresponding TME() function based on these parameters.

[0284] In some examples, TME() may also be pre-stored in the enhanced image generation module, that is, the agreed value between the transcoding end and the encoding end.

[0285] 3) Obtain an intermediate image (denoted as recHDR) based on the tone-mapped base image layer data (baseAfter) and the tone-mapped enhanced image layer data (enhanceAfter).

[0286] Specifically, still referring to Figure 9, the decoding module obtains at least one intermediate image (denoted as recHDR_1...recHDR_N) based on at least one tone-mapped base image (baseAfter_1...baseAfter_N) and at least one tone-mapped enhanced image (enhanceAfter_1...enhanceAfter_N). In the embodiment of the present application, it can also be understood that the tone-mapped base image and the enhanced image are merged to obtain the corresponding intermediate image. It should be noted that this step is only described by taking the generation of a single intermediate image as an example, and this application will not explain them one by one.

[0287] Exemplary methods for obtaining the intermediate image include but are not limited to the following methods:

[0288] Method 1:

[0289] recHDR[i]=baseAfter[i]*enhanceAfter[i]

[0290] Method 2:

[0291] recHDR[i]=baseAfter[i]+enhanceAfter[i]

[0292] Method 3:

[0293] recHDR[i]=f1baseAfter[i]+f2(enhanceAfter[i]).

[0294] Method 4:

[0295] recHDR[i]=f1baseAfter[i]*f2(enhanceAfter[i]).

[0296] Wherein, f1() and f2() are conversion functions in the numerical domain. Exemplarily, the enhanced image generation module may transform the tone mapping enhanced image and / or the tone mapping basic image into the same numerical domain based on the f() function and then merge (including adding or multiplying) them.

[0297] It should be noted that the merging methods involved in the embodiments of the present application are only illustrative examples. In other embodiments, other feasible merging methods can also be adopted, and this application does not limit them.

[0298] Exemplarily, the tone mapping information of the metadata may include information for indicating any merging scheme, and the enhanced image generation module may select a corresponding scheme based on the indication of the metadata.

[0299] In some instances, the merging scheme may also be pre-set in the enhanced image generation module, that is, for the transcoding end and the encoding end, which is not limited in this application.

[0300] In one possible implementation, as described above, the decoded enhanced image and base image may be one or more. In the case where there are multiple enhanced images and / or base images, the enhanced image generation module performs the above processing on the multiple images to obtain recHDR[i]. The specific processing method is as follows:

[0301] Method 1:

[0302] recHDR[i]=A*g(baseAfter1[i]*f1(enchenceAfter1[i]))+B*g(baseAfter2[i]*f2(enchenceAfter2[i]))+C*g(baseAfter3[i]*f3(enchenceAfter3[i]))…

[0303] Among them, f1(), f2() and f3() are conversion functions. The above functions can be included in the tone mapping information in the metadata, or they can be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() can be an adjustment parameter in the tone mapping information of the metadata, or it can be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0304] Method 2:

[0305] recHDR[i]=A*g(baseAfter1[i]*f1(enchenceAfter1[i]))+B*g(baseAfter2[i])+C*g(baseAfter3[i])……,

[0306] Among them, f1() is a conversion function. The above function can be included in the tone mapping information in the metadata, or it can be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() can be an adjustment parameter in the tone mapping information of the metadata, or it can be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0307] Exemplarily, this method can be used to process a scene in which a code stream includes one enhanced image and multiple basic images.

[0308] Method 3:

[0309] recHDR[i]=A*g(baseAfter1[i]+f1(enchenceAfter1[i]))+B*g(baseAfter2[i]+f2(enchenceAfter2[i]))+C*g(baseAfter3[i]+f3(enchenceAfter3[i]))…

[0310] Among them, f1(), f2() and f3() are conversion functions. The above functions can be included in the tone mapping information in the metadata, or they can be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() can be an adjustment parameter in the tone mapping information of the metadata, or it can be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0311] Method 4:

[0312] recHDR[i]=A*g(baseAfter1[i]+f1(enchenceAfter1[i]))+B*g(baseAfter2[i])+C*g(baseAfter3[i])……

[0313] Among them, f1() is a conversion function. The above function can be included in the tone mapping information in the metadata, or it can be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() can be an adjustment parameter in the tone mapping information of the metadata, or it can be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0314] Exemplarily, this method can be used to process a scene in which a code stream includes one enhanced image and multiple basic images.

[0315] Method 5:

[0316] recHDR[i]=A*baseAfter1[i]+A1*f1(enchenceAfter1[i])+baseAfter1[i]+B*baseAfter2[i]+B1*f1(enchenceAfter2[i])+C*baseAfter3[i]+C1*f1(enchenceAfter3[i])……

[0317] Among them, f1(), f2() and f3() are conversion functions. The above functions can be included in the tone mapping information in the metadata, or they can be preset by the transcoding end (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() can be an adjustment parameter in the tone mapping information of the metadata, or it can be preset by the transcoding end, which is not limited in this application. Optionally, A, B, C... etc. are constants preset by the transcoding end, or included in the tone mapping information of the metadata.

[0318] 4) Based on the intermediate image, obtain the composite image (denoted as HDRPic)

[0319] Specifically, as described above, the tone mapping information of the metadata may further include adjustment parameters. The enhanced image generation module may perform numerical adjustments on at least one intermediate image (recHDR_1 . . . recHDR_N) based on the adjustment parameters to obtain a composite image.

[0320] The adjustment method is as follows:

[0321] Method 1:

[0322] HDRPic[i]=recHDR[i]

[0323] In this method, it can also be understood that no numerical adjustment is made, that is, the numerical adjustment amplitude is 0.

[0324] Method 2:

[0325] HDRPic[i]=recHDR[i]+X

[0326] Where X is the adjustment parameter.

[0327] Optionally, X may be an adjustment parameter in the tone mapping information of the metadata, or may be preset by the transcoding end, which is not limited in this application.

[0328] Method 3:

[0329] HDRPic[i]=g(recHDR[i])

[0330] Here, g() is an inverse normalization function, a mapping function, a tone mapping function, or an inverse mapping function.

[0331] Optionally, g() can be an adjustment parameter in the tone mapping information of the metadata, or it can be preset by the transcoder. This application does not limit this. The decoding function can determine the corresponding g() function based on these parameters. Optionally, g() can be a global or local tone mapping function. In this method, the enhanced image generation module can perform global tone mapping or local tone mapping on the enhanced image based on the g() function.

[0332] It should be noted that the numerical adjustment step is optional. If the encoder performs numerical adjustment during the encoding process, the transcoder will perform numerical adjustment accordingly. Conversely, if the encoder does not perform numerical adjustment during the encoding process, the transcoder does not need to perform numerical adjustment. Of course, it can also be understood that the numerical adjustment is performed, but the numerical adjustment amplitude is 0 (i.e., method 1 of numerical adjustment).

[0333] It should be noted that this application does not limit baseAfter[i] and enchenceAfter[i] to any domain, which can be a linear domain or a PQ domain, a LOG domain, etc. This application also does not limit the color space of baseAfter[i] and enchenceAfter[i], which can be a YUV, RGB, Lab, HSV, etc. color space.

[0334] It should be further explained that after the transcoding end obtains the composite image, it can further perform other image processing on the composite image, which is not limited in this application.

[0335] Exemplarily, the enhanced image generation module outputs the first composite image and the first basic image to the image processing module. The image processing module may perform image processing on the first composite image and the first basic image, respectively, according to the target image processing method, to obtain a composite image (e.g., referred to as the second composite image in the embodiment of the present application) and a basic image (e.g., referred to as the second basic image in the embodiment of the present application) after image processing.

[0336] During the image processing process, the image processing module needs to determine the target image processing method. In the embodiment of the present application, the image processing module may determine the target image processing method in any of the following ways, including but not limited to:

[0337] 1) Determined by user settings

[0338] 2) Determine based on metadata

[0339] 3) Determine according to the preset method.

[0340] In the embodiment of the present application, a variety of image processing methods are provided (which will be described in detail below). The specific method selected as the target image processing method can be selected through any of the three methods mentioned above.

[0341] In one example, during the transcoding process, the transcoding end can provide different options, each of which can correspond to one or more image processing methods (the corresponding relationship can be set according to actual needs and is not limited in this application) for the user to choose. The user can select the corresponding image processing method based on the required image quality, thereby achieving controllable image quality after transcoding. Optionally, the user described in the embodiments of the present application can refer to an end user, or a developer or operation and maintenance personnel, etc., which is not limited in this application.

[0342] In the embodiment of the present application, the optional image processing operations include but are not limited to at least one of the following: image scaling, image sharpening, image deblurring, adding watermarks, removing watermarks, etc.

[0343] Exemplarily, image scaling includes image magnification and image reduction. Image reduction can optionally be to reduce the image size (i.e., resolution) so that the size of the reduced image file becomes smaller than before processing, effectively saving the storage cost of the transcoding end and the receiving end, and reducing transmission overhead. For example, if user 1 shares a double-layer picture to a third-party platform, the third-party platform provides transcoding services. Considering the storage cost, the platform may downsample the image (reduce the resolution) before transmitting it to user 2. Conversely, image magnification can optionally be to enlarge the image size (i.e., resolution), and the image file size becomes larger than before processing, which can improve the image quality and further improve the image display effect. For example, after user 1 takes a double-layer image, he drags it into the image editing software (i.e., the transcoding end) for processing. The transcoding end can transcode the double-layer image and perform image processing on the image data during the transcoding process to enlarge the image data and improve the image resolution.

[0344] For example, image sharpening increases the contrast between edge pixels and emphasizes the transition between light and dark areas. This makes the image outline clearer, enhances local contrast, and reveals more details in the image. For example, the transcoding end receives a dual-layer code stream, and the image in the code stream may be unclear because the original image was not imported when it was captured. The user can select the sharpening function provided by the transcoding end, which can sharpen the image to improve image quality. Conversely, image desaturation can optionally reduce the contrast between edge pixels, making the image outline more blurred.

[0345] There are two main purposes for adding watermarks: user demand and the transcoding platform's actions for publicity or security reasons. The goal is to add new logos or content to the image. The transcoding platform can add the designated watermark to the image during image processing. Conversely, watermark removal can be done by removing the watermark from the image.

[0346] In another example, as described above, the tone mapping information in the metadata may include a target processing mode. That is, the image processing module may determine the image processing mode based on the target processing mode carried in the tone mapping information in the metadata.

[0347] In another example, the transcoding end may preset a target image processing method.

[0348] Exemplarily, the image processing module performs an image processing operation on the first composite image and the first basic image based on a target image processing method to obtain a second composite image and a second basic image.

[0349] The present application provides the following image processing methods: The image processing module can perform image processing operations on the first composite image and the first basic image based on the following image processing methods to obtain a third HDR image and a second basic image.

[0350] The target image processing method is one of the following image processing methods. It should be noted that the image processing methods listed in the embodiment of the present application are only for illustrative purposes, and other embodiments may include more image processing methods, which are not limited in the present application.

[0351] Method 1: Perform an image processing operation on the first composite image (e.g., this application may record this as the first image processing operation), and perform an image processing operation on the first basic image (e.g., this application may record this as the second image processing operation). To clearly describe the different image processing operations, the following text distinguishes between the first image processing operation and the second image processing operation, and will not repeat them here.

[0352] In the embodiment of the present application, the image processing operations include but are not limited to: scaling, sharpening, adding watermarks, etc. Of course, the image processing operations listed in the embodiment of the present application are only illustrative examples and can be set according to actual needs, and this application does not limit them.

[0353] That is to say, in method one, the first image processing operation performed by the image processing module on the first composite image may include at least one of scaling, sharpening, etc. Of course, in other embodiments, other image processing methods may also be included, which can be set according to actual needs and is not limited in this application.

[0354] The image processing module performs a first image processing operation on the first composite image, which can also be understood as modifying the first composite image, that is, modifying the image. The modification includes modifying the image resolution, image color, and image content, which will not be repeated below.

[0355] Exemplarily, the image processing module outputs a second composite image.

[0356] Furthermore, the image processing module performs a second image processing operation on the first basic image and outputs a second basic image. A description thereof may refer to the first synthesized image and will not be repeated here.

[0357] In the embodiment of the present application, the first image processing operation and the second image processing operation may be the same. In other embodiments, the first image processing operation and the second image processing operation may also be different, which is not limited in the present application.

[0358] For example, the image processing module can reduce and sharpen the first composite image, and similarly reduce and sharpen the first basic image. This results in a lower quality second composite image and second basic image, but also a lower data size. In this example, the reduced data size of the transcoded image effectively conserves storage resources on the transcoding end and reduces the bandwidth consumed by the bitstream during transmission.

[0359] Method 2: performing the first image processing operation on the first composite image, while not processing the first basic image.

[0360] Exemplarily, the image processing module performs a first image processing operation on the first composite image, which can also be understood as modifying the first composite image, and outputs a second composite image.

[0361] The image processing module does not process the first basic image, and the second basic image output by the image processing module to the encoding module is the first basic image. It can be understood that after the first basic image is input to the image processing module, the second basic image output is the first basic image.

[0362] For other undescribed parts, please refer to Method 1 and will not be repeated here.

[0363] In a possible implementation, some simple processing may be performed on the first basic image, such as scaling, etc., which is not limited in this application.

[0364] Method three: the first composite image is not processed, and the second image processing operation is performed on the first basic image.

[0365] Exemplarily, the image processing module performs the second image processing operation on the first basic image, which can also be understood as modifying the first basic image, and outputting the second basic image.

[0366] The image processing module does not process the first composite image, and the second composite image output by the image processing module to the encoding module is the first composite image. It can be understood that after the first composite image is input to the image processing module, the second composite image output is the first composite image.

[0367] For other undescribed parts, please refer to Method 1 and will not be repeated here.

[0368] In a possible implementation, some simple processing may be performed on the first composite image, such as scaling, etc., which is not limited in this application.

[0369] Method 4: performing a first image processing operation on the first composite image to obtain a second composite image.

[0370] Based on the second composite image, a second basic image is obtained.

[0371] Exemplarily, the image processing module performs a first image processing operation on the first composite image, which can also be understood as modifying the first composite image, and outputs a second composite image.

[0372] The image processing module can obtain a second basic image based on the second composite image. Specifically, the image processing module processes the second composite image based on a preset algorithm to obtain a second basic image. The preset algorithm may include, but is not limited to, a global tone mapping algorithm, a local tone mapping algorithm, or an inverse algorithm of the TMB() function (the concept of this function can be referred to above), etc., and can be set according to actual needs and is not limited in this application. In this way, the output composite image can be consistent in style with the basic image, avoiding backward compatibility issues caused by large style differences.

[0373] Method five: performing a first image processing operation on the first basic image to obtain a second basic image.

[0374] Based on the second basic image, a second composite image is obtained.

[0375] Exemplarily, the image processing module performs the second image processing operation on the first basic image, which can also be understood as modifying the first basic image, and outputting the second basic image.

[0376] The image processing module may obtain a second composite image based on the second basic image. Specifically, the image processing module processes the second basic image based on a preset algorithm to obtain the second composite image. The preset algorithm may include, but is not limited to, a global tone mapping algorithm, a local tone mapping algorithm, or a forward algorithm of the TMB() function (the concept of this function can be found above).

[0377] Method six: performing a first image processing operation on the first basic image to obtain a second basic image.

[0378] A second image processing is performed based on the first enhanced image to obtain a second enhanced image.

[0379] In the embodiment of the present application, the image processing module may perform a second image processing operation on the first basic image, which may also be understood as modifying the first basic image and outputting a second basic image.

[0380] Furthermore, the image processing module may perform a second image processing on the decoded first enhanced image to obtain a second enhanced image. Optionally, the first image processing and the second image processing may be the same or different.

[0381] In this example, the enhanced image generation module may not participate in the transcoding process, that is, the decoding module outputs the first basic image and the first enhanced image to the image processing module, and the image processing module processes the first basic image and the first enhanced image and outputs the second basic image and the second enhanced image.

[0382] Optionally, the second image processing of the first enhanced image by the image processing module may include but is not limited to reducing or enlarging the enhanced image, etc., which is not limited in this application.

[0383] As shown in FIG7B , the transcoding end may obtain a second code stream based on the second composite image and the second basic image. Specifically, the image processing module obtains the second composite image and the second basic image and outputs the second composite image and the second basic image to the enhanced image generation module.

[0384] The enhanced image generation module obtains a second enhanced image based on the second basic image and the second composite image. Optionally, the process executed by the enhanced image generation module in FIG7B may also be executed by the image processing module in some examples, which is not limited in this application.

[0385] The encoding module can encode the second basic image, the second enhanced image and the metadata to obtain the second basic image encoding data (i.e., the third encoding data) and the second enhanced image encoding data (i.e., the fourth encoding data) and the encoded metadata (which can also be called the second metadata encoding data). Exemplarily, the second basic image encoding data is used to characterize the second basic image, and the second enhanced image encoding data is used to characterize the second enhanced image. Optionally, the encoding method can use any encoding method such as H.265, H.264, etc., which can be set according to actual needs and is not limited in this application.

[0386] The encapsulation module may encapsulate the second basic image coded data, the second enhanced image coded data, and the metadata coded data to obtain a second code stream, which may also be referred to as a second dual-layer code stream. The second code stream includes, but is not limited to, the second basic image coded data, the second enhanced image coded data, and the metadata coded data (which may also be referred to as second metadata coded data). Optionally, the metadata coded data may also be transmitted via a separate code stream, which is not limited in this application.

[0387] In this way, the second composite image, which is composed of the second base image and the second enhanced image in the second dual-layer codestream, differs from the first composite image according to the image processing operation. For example, if the image processing operation is scaling the image data, the difference between the first and second composite images is the difference in resolution. If the image processing operation is sharpening or blunting the image data, the difference between the first and second composite images is the difference in clarity. For example, the second composite image has higher image quality and clearer details than the first composite image. If the image processing operation is adding or removing a watermark, the difference between the first and second composite images can optionally be that the second composite image has more or less watermark than the first composite image.

[0388] This application provides three encoding methods:

[0389] Method 1:

[0390] 1. Encode the second basic image to obtain second basic image encoded data. If there are multiple second basic images, encode each second basic image to obtain multiple second basic images.

[0391] 2. Encode the second enhanced image to obtain second enhanced image encoded data. If there are multiple second enhanced images, encode the multiple second enhanced images to obtain multiple second enhanced images. In some embodiments, a preset encoding condition (which can be set according to actual needs) can be set. Multiple second enhanced images that meet the preset encoding condition can be encoded to obtain at least one second enhanced image, and other enhanced images that do not meet the preset encoding condition can be left unprocessed.

[0392] Method 2:

[0393] 1. Encode the second basic image to obtain second basic image encoded data.

[0394] 2. Decode the second basic image coded data to obtain a decoded basic image.

[0395] 3. Obtain the difference (or gain) between each pixel of the decoded basic image and the second synthesized image.

[0396] 4. Encode the difference value to obtain second enhanced image encoded data.

[0397] Method 3:

[0398] 1. Encode the second basic image to obtain second basic image encoded data.

[0399] 2. Obtain the difference (or gain) between the second basic image coded data and each pixel of the second synthesized image.

[0400] 3. Encode the difference value to obtain second enhanced image encoded data.

[0401] S803: Output a second double-layer code stream, where the second double-layer code stream includes the third coded data and the fourth coded data.

[0402] Exemplarily, the transcoding end outputs a second double-layer code stream. The second double-layer code stream includes basic image coding data (i.e., third coding data) and enhanced image coding data (i.e., fourth coding data). The transcoding end can output the second double-layer code stream to the receiving end for display, or output it to a storage device (which can be local storage or independent storage) for storage, which is not limited in this application.

[0403] The embodiment of the present application further provides an image processing device 1000 , see FIG10 , which is an example diagram of an image processing device provided in the embodiment of the present application, and is used to implement the method in the embodiment of the present application.

[0404] As shown in FIG10 , the image processing apparatus 1000 may include a processor 1001 for executing a program or instruction stored in a memory 1002 . When the program or instruction stored in the memory 1002 is executed, the processor 1001 is used to execute the image encoding method in the embodiment shown in FIG8 .

[0405] Optionally, the image processing apparatus 1000 may further include a communication interface 1003. FIG10 uses a dotted line to indicate that the communication interface 1003 is optional for the image processing apparatus 1000.

[0406] Among them, the number of processors 1001, memories 1002 and communication interfaces 1003 does not constitute a limitation on the embodiments of the present application. During specific implementation, they can be arbitrarily configured according to business requirements.

[0407] Optionally, the memory 1002 is located outside the image processing device 1000 .

[0408] Optionally, the image processing apparatus 1000 includes the memory 1002, which is connected to the at least one processor 1001 and stores instructions executable by the at least one processor 1001. FIG10 uses dashed lines to indicate that the memory 1002 is optional for the image processing apparatus 1000.

[0409] The processor 1001 and the memory 1002 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0410] The specific connection medium between the processor 1001, memory 1002, and communication interface 1003 is not limited in the embodiments of the present application. In Figure 10, the processor 1001, memory 1002, and communication interface 1003 are connected via a bus 1004. The bus is represented by a bold line in Figure 10. The connection methods between other components are only for schematic illustration and are not limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bold line is used in Figure 10, but this does not mean that there is only one bus or one type of bus.

[0411] It should be understood that the processors mentioned in the embodiments of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that is implemented by reading software code stored in a memory.

[0412] Exemplarily, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0413] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0414] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0415] It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0416] An embodiment of the present application further provides a computer-readable storage medium, comprising a program or instructions, which enables the method in FIG8 to be executed when the program or instructions are run on a computer.

[0417] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0418] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.

[0419] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0420] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0421] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is intended to include such modifications and variations.

Claims

1. An image processing method, characterized in that: include: Acquire a first double-layer code stream, the first double-layer code stream includes first coded data and second coded data, the first coded data is used to represent a first basic image, the second coded data is used to represent a first enhanced image, and the first basic image and the first enhanced image are used to synthesize a first synthesized image; performing an image processing operation based on the first double-layer code stream to obtain third coded data and fourth coded data, wherein the third coded data is used to represent a second basic image, the fourth coded data is used to represent a second enhanced image, the second basic image and the second enhanced image are used to synthesize a second synthesized image, and there is a difference associated with the image processing operation between the second synthesized image and the first synthesized image; A second double-layer code stream is output, where the second double-layer code stream includes the third coded data and the fourth coded data.

2. The method according to claim 1, characterized in that The method further comprises: Decoding the first coded data to obtain the first basic image, and decoding the second coded data to obtain the first enhanced image; The first basic image and the first enhanced image are combined to obtain the first composite image.

3. The method according to claim 2, characterized in that The performing of an image processing operation based on the first double-layer code stream to obtain third coded data and fourth coded data includes: performing the image processing operation on the first composite image to obtain the second composite image; Based on the second composite image, acquiring a second basic image; The third encoded data and the fourth encoded data are obtained based on the second synthesized image and the second basic image.

4. The method according to claim 2, characterized in that: The performing of an image processing operation based on the first double-layer code stream to obtain third coded data and fourth coded data includes: performing the image processing operation on the first composite image to obtain the second composite image; Performing the image processing operation on the first basic image to obtain a second basic image; The third encoded data and the fourth encoded data are obtained based on the second synthesized image and the second basic image.

5. The method according to claim 3 or 4, characterized in that: The obtaining the third coded data and the fourth coded data based on the second synthesized image and the second basic image comprises: obtaining the second enhanced image based on the second synthesized image and the second basic image; encoding the second basic image to obtain the third encoded data; The second enhanced image is encoded to obtain the fourth encoded data.

6. The method according to claim 3 or 4, characterized in that: The obtaining the third coded data and the fourth coded data based on the second synthesized image and the second basic image comprises: encoding the second basic image to obtain the third encoded data; The fourth encoded data is obtained based on the third encoded data and the second synthesized image.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: An operation mode of the image processing operation is obtained, where the operation mode includes at least one of the following: zooming in, zooming out, sharpening, blunting, adding a watermark, or removing a watermark.

8. The method according to claim 7, characterized in that The operation mode is preset, or the operation mode of obtaining the image processing operation includes: Determine the operation mode based on the received user operation or user demand; or, The operation mode is obtained from the first dual-layer code stream.

9. An image processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 8.

11. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 8.

Images