Image display method and device

Abstract

The invention provides an image display method and device. The method comprises the following steps: acquiring a double-layer code stream comprising first coded data, second coded data and metadata; wherein the first coded data is used for representing the basic image, and the second coded data is used for representing the enhanced image. And acquiring a basic image and an enhanced image based on the first encoded data and the second encoded data. And combining the basic image and the enhanced image to obtain a composite image. And performing tone mapping on the composite image based on first tone mapping information to obtain a first target image, the first tone mapping information being determined based on the metadata and first display information of the electronic device. And displaying the first target image. According to the image display method suitable for the double-layer code stream, the tone mapping information suitable for the current display capability of the electronic equipment can be determined based on the display information of the electronic equipment, so that the target image meeting the current display requirement of the electronic equipment is obtained, the display effect of the electronic equipment is improved, and the visual experience of a user is improved.

Description

Image display method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 15, 2023, with application number 202311034406.7 and application name “Image Processing Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to image processing technology, and in particular to an image display method and device. Background Art

[0003] Currently, the image processing process generally includes front-end processing (e.g., image generation and encoding), intermediate processing (e.g., transcoding and re-encoding), and back-end processing (e.g., decoding and displaying images). The aforementioned image processing process is mostly for single-layer bitstreams, such as those derived from standard dynamic range (SDR) images. However, for high dynamic range (HDR) images, the single-layer bitstream processing method is not applicable.

[0004] Summary of the Invention

[0005] The present application provides an image display method and device, which can be applied to image display scenarios with dual-layer code streams.

[0006] In a first aspect, the present application provides an image display method. The method includes: an electronic device obtaining a dual-layer code stream, the dual-layer code stream comprising first coded data, second coded data, and metadata, the first coded data representing a base image, and the second coded data representing an enhanced image. Based on the first and second coded data, the electronic device obtains a base image and an enhanced image. The electronic device merges the base image and the enhanced image to obtain a composite image. The electronic device performs tone mapping on the composite image based on first tone mapping information to obtain a first target image, the first tone mapping information being determined based on the metadata and first display information of the electronic device. The electronic device displays the first target image. Thus, by synthesizing the base image and the enhanced image after decoding, a single composite image is obtained that fits into a single buffer, resolving the inapplicability of single-layer code stream processing methods. A dual-layer code stream, after decoding, typically contains two image buffers, a base image and an enhanced image. This application separates the decoding, processing, and display of the dual-layer code stream. After decoding, the two images are merged into a single composite image, merging the buffers for the two images into a single buffer. In this way, existing module interfaces can be reused from decoding to image acquisition and then to display. During display, tone mapping can be performed based on the display-related information conveyed by the metadata, ultimately achieving the correct processing effect. This avoids the trouble of redesigning the application due to the inability to use the original interface. Furthermore, this application can determine the corresponding tone mapping information based on the metadata and the current display information of the electronic device, thereby generating a target image that adapts to the current display capabilities of the electronic device, thereby improving the display effect of the electronic device and enhancing the user's visual experience.

[0007] In one possible implementation, the display information includes at least one of the following: display information of the display screen and display environment information. Thus, the electronic device can determine corresponding tone mapping parameters based on the current display capabilities of the display screen and the environment surrounding the display screen, thereby enabling the generated target image to adapt to the current display capabilities of the display screen and the environment surrounding the display screen.

[0008] In one possible implementation, the display information of the display includes at least one of the following: a brightness range of the display and a size of the display. Thus, the electronic device can determine corresponding tone mapping information based on the display information, such as the brightness and size of the display, so that the generated target image can adapt to the current display capabilities of the display and the surrounding environment.

[0009] In one possible implementation, the display environment information includes at least one of the following: the brightness of the electronic device's surrounding environment and the color temperature of the surrounding environment. Thus, the electronic device can determine corresponding tone mapping information based on parameters such as the ambient brightness and color temperature, thereby enabling the generated target image to adapt to the current display capabilities of the display screen and the surrounding environment.

[0010] In one possible implementation, the metadata includes at least one tone mapping information and at least one preset display information corresponding to the at least one tone mapping information. Before performing tone mapping on the composite image based on the first tone mapping information, the method further includes: determining, based on the display information of the electronic device, target preset display information that matches the display information of the electronic device; and adjusting the tone mapping information corresponding to the target preset display information based on a difference between the target preset display information and the display information of the electronic device to obtain the first tone mapping information. In this way, the electronic device can obtain the tone mapping information corresponding to the current display information of the electronic device using the tone mapping information carried in the metadata, thereby enabling the generated target image to adapt to the current display capabilities of the electronic device and improve the display effect.

[0011] In one possible implementation, the metadata includes at least one tone mapping information and at least one preset display information corresponding to the at least one tone mapping information. Before performing tone mapping on the composite image based on the first tone mapping information, the method further includes: adjusting the at least one tone mapping information based on a difference between the electronic device's display information and the at least one preset display information to obtain the first tone mapping information. In this way, the electronic device can use the tone mapping information carried in the metadata to obtain the tone mapping information corresponding to the electronic device's current display information, thereby enabling the generated target image to adapt to the electronic device's current display capabilities and improving display quality.

[0012] In one possible implementation, the method further includes: performing tone mapping on the synthesized image based on second tone mapping information to obtain a second target image; wherein the second tone mapping information is determined based on the metadata and second display information of the electronic device; and displaying the second target image. In this way, if the display information of the electronic device changes dynamically, the electronic device can dynamically adjust the tone mapping information based on the changed display information, thereby ensuring that the generated target image adapts to the current display capabilities of the electronic device.

[0013] In one possible implementation, before merging the base image and the enhanced image, the method further includes determining an allowable range of hue distortion. In this way, the electronic device can execute corresponding image display processes based on different scene requirements to improve adaptability to different scenes.

[0014] In a possible implementation, merging the basic image and the enhanced image includes: merging the basic image and the enhanced image to obtain a composite image when the allowable range of the hue distortion is within a preset range.

[0015] In a second aspect, the present application provides an image display device, comprising: an acquisition module for acquiring a dual-layer code stream, the dual-layer code stream comprising first coded data, second coded data, and metadata, the first coded data being used to represent a basic image, and the second coded data being used to represent an enhanced image. A decoding module for acquiring a basic image and an enhanced image based on the first coded data and the second coded data. A synthesis module for merging the basic image and the enhanced image to obtain a synthesized image. A tone mapping module for performing tone mapping on the synthesized image based on first tone mapping information to obtain a first target image, the first tone mapping information being determined based on metadata and first display information of the electronic device. A display module for displaying the first target image.

[0016] In a possible implementation, the display information of the electronic device includes at least one of the following: display information of a display screen and display environment information.

[0017] In a possible implementation, the display information of the display screen includes at least one of the following: a brightness range of the display screen and a size of the display screen.

[0018] In a possible implementation, the display environment information includes at least one of the following: brightness of the surrounding environment of the electronic device and color temperature of the surrounding environment.

[0019] In one possible implementation, the metadata includes at least one tone mapping information and at least one preset display information corresponding to the at least one tone mapping information. The tone mapping module is specifically used to determine, based on the display information of the electronic device, target preset display information that matches the display information of the electronic device; and adjust the tone mapping information corresponding to the target preset display information based on the difference value between the target preset display information and the display information of the electronic device to obtain first tone mapping information.

[0020] In one possible implementation, the metadata includes at least one tone mapping information and at least one preset display information corresponding to the at least one tone mapping information. The tone mapping module is specifically used to adjust the at least one tone mapping information based on the difference value between the display information of the electronic device and the at least one preset display information to obtain the first tone mapping information.

[0021] In one possible implementation, the tone mapping module is further configured to perform tone mapping on the composite image based on second tone mapping information to obtain a second target image; wherein the second tone mapping information is determined based on the metadata and second display information of the electronic device. The display module is further configured to display the second target image.

[0022] In a possible implementation, the apparatus further includes a determination module configured to determine an allowable range of hue distortion.

[0023] In a possible implementation, the synthesis module is further configured to merge the basic image and the enhanced image to obtain a synthesized image when the allowable range of the hue distortion is within a preset range.

[0024] In a third aspect, the present application provides an encoding method, comprising: obtaining a base image and an enhanced image; encoding the base image and the enhanced image to obtain first encoded data and second encoded data; and outputting a dual-layer codestream, the dual-layer codestream including the first encoded data, the second encoded data, and metadata.

[0025] In a fourth aspect, the present application provides an electronic device comprising one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by one or more processors, the one or more processors implement any of the methods described in the first aspect above.

[0026] In a fifth aspect, the present application provides a computer-readable storage medium comprising a computer program, wherein when the computer program is executed on a computer, the computer executes the method according to any one of the above-mentioned first aspects.

[0027] In a sixth aspect, the present application provides a computer program, which, when executed by a computer, is used to perform any of the methods described in the first aspect above.

[0028] In a seventh aspect, the present application also provides a computer program product, which includes computer program code. When the computer program code runs on a computer, it enables the computer to execute the operations and / or processing performed by the electronic device in any of the above method embodiments.

[0029] In an eighth aspect, an embodiment of the present application provides a code stream structure, which includes encoded data of a basic image, encoded data of an enhanced image, and metadata, wherein the metadata includes tone mapping information. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] FIG1 is a schematic block diagram of a coding solution system used in an embodiment of the present application;

[0031] FIG2A is a block diagram illustrating an exemplary content providing system;

[0032] FIG2B is a schematic structural diagram of an exemplary terminal device;

[0033] FIG3A is a schematic diagram illustrating an exemplary workflow of a streaming media system;

[0034] FIG3B is a schematic diagram illustrating an exemplary streaming media system architecture;

[0035] FIG4A is an exemplary architecture diagram of an image processing system;

[0036] FIG4B is an exemplary architecture diagram of an image processing system;

[0037] FIG5 is a schematic diagram exemplarily illustrating dynamic mapping of a high dynamic range in the real world to a low dynamic range of a display device;

[0038] FIG6 is a schematic diagram showing the structure of a receiving end;

[0039] FIG7 is a schematic flow chart illustrating an exemplary image display method;

[0040] FIG8 is a schematic flow chart illustrating an exemplary image display method;

[0041] FIG9 is a schematic diagram illustrating an exemplary tone mapping process;

[0042] FIG10 is a schematic flow chart illustrating an exemplary image display method;

[0043] FIG. 11 is a schematic structural diagram illustrating an exemplary image display device. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0045] The terms "first," "second," and the like in the description, embodiments, claims, and drawings of this application are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or order. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions, such as, for example, inclusion of a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0047] The following are the related technologies involved in this application:

[0048] In order to solve the above technical problems, the present application provides an image display method and device, and the technical solution of the present application will be described below.

[0049] FIG1 is a schematic block diagram of a video encoding and decoding system used in embodiments of the present application. The video encoder 20 (or simply encoder 20 ) and video decoder 30 (or simply decoder 30 ) in the video encoding and decoding system 10 represent devices that can be used to perform various techniques according to the various examples described in this application.

[0050] As shown in FIG1 , a coding and decoding system 10 includes a source device 12 , which is configured to provide coded data 21 such as coded images to a destination device 14 for decoding the coded data.

[0051] The source device 12 includes an encoder 20 and, optionally, may further include an image source 16 , an image preprocessor 18 (or a preprocessing unit), and a communication interface or a communication unit 22 .

[0052] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0053] In order to distinguish the processing performed by the pre-processor 18 or the pre-processing unit 18 , the image or image data 17 may also be referred to as a raw image or raw image data 17 .

[0054] The preprocessor 18 is configured to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.

[0055] The video encoder 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 .

[0056] Communication interface 22 in source device 12 may be used to receive encoded image data 21 and transmit the encoded image data 21 (or any other processed version thereof) to another device, such as destination device 14, or any other device, via communication channel 13 for storage or direct reconstruction.

[0057] Destination device 14 includes a decoder 30 (eg, video decoder 30 ) and, optionally, a communication interface or communication unit 28 , a post-processor 32 (or post-processing unit 32 ), and a display device 34 .

[0058] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, a storage device that is an encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0059] The communication interface 22 and the communication interface 28 may be used to send or receive the encoded image data 21 or the encoded data via a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any type of network, such as a wired network, a wireless network, or any combination thereof, any type of private network and public network, or any combination thereof.

[0060] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0061] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21 .

[0062] Both the communication interface 22 and the communication interface 28 can be configured as a unidirectional communication interface as indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 in Figure 1, or a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0063] The decoder 30 is configured to receive the encoded image data 21 and provide decoded image data 31 or a decoded image 31 .

[0064] The post-processor 32 in the destination device 14 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.

[0065] A display device 34 in the destination device 14 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0066] Although FIG1 shows source device 12 and destination device 14 as separate devices, device embodiments may also include both source device 12 and destination device 14 or the functionality of both source device 12 and destination device 14, that is, both source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality. In these embodiments, source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality may be implemented using the same hardware and / or software or through separate hardware and / or software, or any combination thereof.

[0067] Based on the description, it will be apparent to those skilled in the art that the existence and (precise) division of different units or functions in the source device 12 and / or the destination device 14 shown in FIG. 1 may vary depending on actual devices and applications.

[0068] FIG2A is a block diagram of an exemplary content provision system. The following describes the content provision system for the content distribution service used in this application in conjunction with FIG2A . The content provision system 20 includes a capture device 2102, a terminal device 2106, and an (optional) display 2126. The capture device 2102 communicates with the terminal device 2106 via a communication link 2104. The communication link may include the aforementioned communication channel 13. The communication link 2104 includes, but is not limited to, WiFi, Ethernet, wired, wireless (3G / 4G / 5G), USB, or any combination thereof.

[0069] Capture device 2102 generates data and encodes the data. Alternatively, capture device 2102 can distribute the data to a streaming media server (not shown), which encodes the data and transmits the encoded data to terminal device 2106. Capture device 2102 includes but is not limited to a camera, a smart phone or tablet computer, a computer or laptop computer, a video conferencing system, a PDA, a vehicle-mounted device, or any combination thereof. For example, capture device 2102 can include the above-mentioned source device 12. When the data includes video, the encoder 20 in capture device 2102 can actually perform video encoding processing. When the data includes audio (i.e., sound), the encoder 20 in capture device 2102 can actually perform audio encoding processing. In some actual scenarios, capture device 2102 distributes encoded video data and encoded audio data by multiplexing the encoded video data and the encoded audio data together. In other actual scenarios, such as in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. Capture device 2102 distributes the encoded audio data and the encoded video data to terminal device 2106 respectively.

[0070] The terminal device 2106 in the content providing system 20 receives and regenerates the encoded data. The terminal device 2106 can be a device with data reception and recovery capabilities, such as a smartphone or tablet 2108, a computer or laptop 2110, a network video recorder (NVR) / digital video recorder (DVR) 2112, a television 2114, a set-top box (STB) 2116, a video conferencing system 2118, a video surveillance system 2120, a personal digital assistant (PDA) 2122, an in-vehicle device 2124, or any combination thereof, or any other device capable of decoding the encoded data. For example, the terminal device 2106 can include the destination device 14 described above. When the encoded data includes video, the video decoder 30 in the terminal device prioritizes video decoding. When the encoded data includes audio, the audio decoder in the terminal device prioritizes audio decoding. The terminal device 2106 can be a video playback application, a streaming media playback application, a streaming media playback platform, or a live broadcast platform running on the terminal device.

[0071] For end devices with displays, such as smartphones or tablets 2108, computers or laptops 2110, NVRs / DVRs 2112, televisions 2114, PDAs 2122, or in-vehicle devices 2124, the end device can send the decoded data to its display. For end devices without displays, such as STBs 2116, video conferencing systems 3118, or video surveillance systems 2120, the device can be connected to an external display 2126 to receive and display the decoded data.

[0072] When performing encoding or decoding, each device in this system may use the image encoding device or image decoding device shown in the above embodiments.

[0073] FIG2B is a schematic diagram of an example structure of the terminal device 2106 in FIG2A . After the terminal device 2106 receives the bitstream from the capture device 2102, the protocol processing unit 2202 analyzes the transmission protocol of the bitstream. The protocol includes, but is not limited to, Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP), or any combination thereof.

[0074] After processing the stream, protocol processing unit 2202 generates a stream file. This file is output to demultiplexing unit 2204. Demultiplexing unit 2204 can separate the multiplexed data into encoded audio data and encoded video data. As mentioned above, in other practical scenarios, such as video conferencing systems, the encoded audio data and encoded video data are not multiplexed. In this case, the encoded data is not transmitted to video decoder 2206 and audio decoder 2208 via demultiplexing unit 2204.

[0075] Through the demultiplexing process, a video elementary stream (ES), an audio ES, and optional subtitles are generated. The video decoder 2206, including the video decoder 30 described in the above embodiment, decodes the video ES by the decoding method shown in the above embodiment to generate video frames, and sends the data to the synchronization unit 2212. The audio decoder 2208 decodes the audio ES to generate audio frames, and sends the data to the synchronization unit 2212. Alternatively, the video frames can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212. Similarly, the audio frames can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212.

[0076] Synchronization unit 2212 synchronizes video frames and audio frames and provides the video / audio to video / audio display 2214. For example, synchronization unit 2212 synchronizes the presentation of video and audio information. The information can be encoded in the syntax using timestamps associated with the representation of the encoded audio and visual data and timestamps associated with the transmission of the data stream.

[0077] If subtitles are included in the bitstream, the subtitle decoder 2210 decodes the subtitles, synchronizes them with the video frames and audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 2216.

[0078] The present invention is not limited to the above-mentioned system. The image encoding device or the image decoding device in the above-mentioned embodiments can be used in other systems such as automobiles.

[0079] FIG3A is a schematic diagram of a workflow of a streaming media system applicable to an embodiment of the present application. The streaming media system applicable to an embodiment of the present application is described below with reference to FIG3A.

[0080] The streaming media system includes a content creation module that generates the required content data, such as video or audio. The streaming media system also includes a video encoding module that encodes the generated content through an encoder. The streaming media system also includes a video stream transmission module that transmits the encoded video in the form of a code stream. Optionally, the video stream transcoding module can convert the format of the video stream into a code stream format of a commonly used transmission protocol for OTT (over-the-top) devices, for example, the protocol includes but is not limited to Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG HTTP Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP) or any combination thereof. Optionally, the video stream storage module can store the video stream and store the original format of the video stream and / or the converted multiple code stream formats for easy use. Furthermore, the streaming system also includes a video stream encapsulation module for encapsulating the video stream to generate an encapsulated video stream. The encapsulated video stream can be referred to as a video streaming package. Exemplarily, the video streaming package can be generated based on a transcoded video stream or a stored video stream. Furthermore, the streaming system also includes a content distribution network (CDN) for distributing the video streaming package to multiple OTT devices, such as mobile phones, computers, tablets, and home projectors.

[0081] It should be noted that video encoding, video streaming transmission, video stream transcoding, video stream storage, video streaming package generation and content distribution network can all be implemented on cloud servers.

[0082] FIG3B is a schematic diagram of an exemplary streaming media system architecture. An exemplary streaming media system architecture of the present application will be described below in conjunction with FIG3B . The streaming media system architecture includes: a client device, a content distribution network, and a cloud server.

[0083] The user on the client device sends a play or playback request to the cloud platform. Optionally, the content of the request can be the title of the movie or TV program to be played.

[0084] The cloud platform makes a decision and responds to the client, sending the client the address of the requested content on the CDN. Optionally, the content sent to the client can be a URL link (uniform resource locator). Specifically, the playback application service in the cloud platform checks user authorization and permissions, and then considers individual client characteristics and current network conditions to determine which specific files are needed to process the playback request. It should be noted that the content delivery network (CDN) regularly reports its operating status, learned routes, and available content (files) to the cache control service in the cloud platform.

[0085] The client then requests the CDN to play the content based on the address, and the CDN provides the content to the client, ultimately completing the client's request.

[0086] Figure 4A is a schematic diagram of a possible system architecture applicable to an embodiment of the present application. The system architecture applicable to the embodiment of the present application is described below in conjunction with Figure 4A. The system architecture of the embodiment of the present application includes: a front-end device, a transmission link, and a terminal display device.

[0087] Among them, the front-end equipment is used to capture or produce HDR / SDR content (for example, HDR / SDR video or images).

[0088] In a possible embodiment, the front-end device can also be used to extract corresponding metadata from the HDR content, which may include global mapping information, local mapping information, and dynamic metadata and static metadata corresponding to the HDR content. The front-end device can send the HDR content and metadata to the terminal display device through a transmission link. Specifically, the HDR content and metadata can be transmitted in the form of a data packet (i.e., an HDR code stream), or respectively in two data packets (i.e., an HDR code stream and a metadata stream), which is not specifically limited in the embodiments of the present application.

[0089] Optionally, the terminal display device can be used to receive metadata and HDR content, and extract the global mapping information, local mapping information, and the terminal display device information contained in the corresponding metadata according to the HDR content, obtain a mapping curve to perform global tone mapping and local tone mapping on the HDR content, convert it into display content adapted for the HDR display device or SDR device in the terminal display device, and display it. It should be understood that in different embodiments, the terminal display device may include a display device with a display capability of a lower dynamic range or a higher dynamic range than the HDR content generated by the front-end device, and this application is not limited thereto.

[0090] Optionally, the front-end device and the terminal display device in this application can be independent and different physical devices. For example, the front-end device can be a video capture device or a video production device, where the video capture device can be a video camera, a camera, an image rendering machine, etc. The terminal display device can be a device with video playback function, such as virtual reality (VR) glasses, a mobile phone, a tablet, a television, a projector, etc.

[0091] Optionally, the transmission link between the front-end device and the terminal display device can be a wireless connection or a wired connection, wherein the wireless connection can adopt technologies such as long term evolution (LTE), fifth generation (5G) mobile communications, and future mobile communications. Wireless connections can also include wireless fidelity (WiFi), Bluetooth, near field communication (NFC), and other technologies. Wired connections can include Ethernet connections, local area network connections, etc. There is no specific limitation on this.

[0092] The present application may also integrate the functions of the front-end device and the terminal display device into the same physical device, for example, a terminal device such as a mobile phone or tablet with a video capture function. The present application may also integrate some functions of the front-end device and some functions of the terminal display device into the same physical device. This is not specifically limited.

[0093] FIG4B is a schematic diagram illustrating a system 100 for applying a method for processing HDR video (or HRD image). As shown in FIG4B , an end-to-end HDR processing system includes, but is not limited to, a transmitter (also referred to as an encoder) and a receiver (also referred to as a decoder or display). Optionally, a transcoding end is also included. The transmitter includes, but is not limited to, an HDR preprocessing module 110 and an HDR video encoding module 120. The decoder includes, but is not limited to, an HDR video decoding module 130 and an HDR Dynamic™ module 140.

[0094] HDR video is input to the HDR preprocessing module 110. This HDR video can be captured by a capture device or processed by a colorist or device using an algorithm. The pixel values ​​in this HDR video have undergone photoelectric transfer, for example, all values ​​in the PQ domain. For example, the dynamic range of a capture device under specific shooting conditions is typically fixed. To obtain an image with a higher dynamic range, multiple images with different exposures are typically captured at the same time, and then synthesized to obtain an image with a higher dynamic range. The resulting high dynamic range image typically has a bit width greater than 10 bits, thus accommodating scenes with a higher dynamic range.

[0095] The HDR pre-processing module 110 is responsible for extracting static metadata and dynamic metadata. If the HDR video uses Director Mode, the HDR pre-processing module 110 calculates the curve parameters of the Director Mode's tone mapping (TM), which will be written into the metadata.

[0096] The HDR video encoding module 120 encodes the HDR video and metadata, and the metadata is embedded in the user-defined part of the code stream. The HDR video encoding module 120 can use any standard encoder, such as High Efficiency Video Coding (HEVC) or the second-generation digital audio and video coding technology standard (Audio Video coding Standard2, AVS2). After the HDR video encoding module completes the encoding, the encoding end can transmit the code stream to the decoding end. For example, in the widely supported HEIF, H.265, and H.266, 10-bit encoding is supported. In the widely supported JPEG and H.264, 8-bit encoding is supported, which does not support HDR encoding well.

[0097] Optionally, in some instances, the encoding end may transmit the code stream to the transcoding end. The transcoding end processes the code stream and saves it or transmits it to the decoding end. Optionally, the encoding end and the transcoding end may be provided on different devices or on the same device (for example, both provided on the transmitting end), which is not specifically limited in this application.

[0098] For example, after the HDR video is decoded, it is encoded using an encoder, which is the transcoding process currently used by applications and services for video distribution through servers.

[0099] After decoding HDR images and videos, frame-by-frame HDR pixel values, along with HDR format identifiers and information, are obtained. HDR images are typically sent to an image processing module for operations such as upsampling and downsampling, as well as enhancement. These processed HDR pixel values, along with HDR format identifiers and information, are then sent to the encoder for re-encoding.

[0100] The HDR video decoding module 130 at the decoding end (ie, the receiving end) decodes the code stream according to the standard corresponding to the code stream format, and outputs HDR decoded video and HDR metadata.

[0101] Based on the decoded HDR video and metadata, as well as the peak brightness of the display device, the HDR Dynamic TM module 140 calculates the TM curve, performs tone mapping, and outputs it to the display. If the HDR video uses Director Mode, the HDR Dynamic TM module 140 does not need to calculate the curve and directly uses the curve parameters in the metadata.

[0102] Before the HDR video is input to the HDR pre-processing module 110, the pixel values ​​of the HDR video need to be photoelectrically transferred. This is mainly because the maximum brightness information of the display device cannot reach the brightness information of the real world, and we browse images through the display device, so the photoelectric transfer function is needed.

[0103] Optionally, based on the system shown in Figure 4A or 4B, obtaining a better experience of HDR images is an end-to-end process, from the production end (corresponding to the above-mentioned sending end, generation, encoding), to the distribution end (corresponding to the above-mentioned transcoding end, transcoding, decoding, and re-encoding), and then to the terminal device (corresponding to the above-mentioned receiving end, decoding and display) to meet the HDR requirements.

[0104] 1) Generation of HDR images

[0105] Capture devices (such as video cameras and still cameras) typically have a fixed dynamic range under specific shooting conditions. To obtain images with a higher dynamic range, multiple images with different exposures can be captured simultaneously and synthesized to create images with a higher dynamic range. The resulting high-dynamic-range images typically have a bit width greater than 10 bits, allowing them to accommodate scenes with a higher dynamic range.

[0106] 2) Encoding and decoding of HDR images and videos

[0107] Among the widely supported HEIF, H.265, and H.266, 10-bit encoding is supported, which can well support HDR encoding.

[0108] 3) Distribution and transcoding of HDR images and videos

[0109] After being decoded and processed, the HDR image is then encoded using an encoder. This is the transcoding process currently used by applications and services when distributing images through servers.

[0110] After decoding, the HDR image obtains frame-by-frame HDR pixel values ​​and HDR format identifiers and information; the HDR image usually needs to be sent to the image processing module to complete image upsampling, downsampling, enhancement and other operations; the processed HDR pixel values ​​and HDR format identifiers and information are then sent to the encoder for re-encoding.

[0111] After decoding, the SDR image obtains SDR pixel values ​​frame by frame; the SDR image usually needs to be sent to the image processing module to complete image upsampling, downsampling, enhancement and other operations; the processed SDR pixel values ​​are then sent to the encoder for re-encoding.

[0112] 4) Decoding and display at the receiving end

[0113] After receiving the code stream, the receiving end calls the system decoding interface to obtain the decoded pixel value buffer; the pixel value buffer passes through the graphics path to obtain the pixel value buffer to be displayed; by calling the system display interface, the pixel value buffer to be displayed is configured to the display hardware, and the display hardware will display according to the status of the display device, including remapping of the dynamic range (which can also be called tone mapping in the embodiment of this application).

[0114] The following is an explanation of the technical terms that may be involved in this application:

[0115] 1. Dynamic range

[0116] Dynamic range is used in many fields to express the ratio of the maximum and minimum values ​​of a variable. In digital images, dynamic range is used to express the ratio between the maximum and minimum grayscale values ​​within the image's displayable range.

[0117] In nature, the brightness of a night scene under a starry sky is about 0.001cd / m 2 , the sun itself is as bright as 10 9 cd / m 2 , the dynamic range can reach 10 9 / 0.001=10 12 cd / m 2 However, in the real world, the brightness of the sun and the brightness of the stars are not obtained at the same time. Therefore, for the same scene in the real world, the dynamic range is usually around 10 -3 cd / m 2 to 10 6 cd / m 2In most current color digital images, the grayscale of the red (R), green (G), and blue (B) channels are each stored using one byte (8 bits). That is, the grayscale range of the R, G, and B channels is 0 to 255, which is the dynamic range of the image, called low dynamic range (LDR).

[0118] cd / m 2 It can also be expressed as nit, which is a unit of light.

[0119] 2. Optical-electro transfer function (OETF)

[0120] Figure 5 is an example of dynamic range mapping in the real-world imaging process. The imaging process of a digital camera is actually a mapping of the high dynamic range of the real world to the low dynamic range of the digital image. As shown in Figure 1, in addition to the brightness of starlight and the sun, the brightness of moonlight is 1cd / m 2 , the brightness of indoor lighting is 100cd / m 2 , the brightness of outdoor cloudy weather is 500cd / m 2 The brightness of outdoor light on a sunny day is 2000cd / m 2 The real-world brightness range is 100cd / m 2 to 2000cd / m 2 The brightness range corresponding to the storage method of the display device is 1cd / m 2 to 200cd / m 2 Into a mapping relationship.

[0121] Since the storage method of the display device cannot achieve the high brightness of the real world, the electro-optical transfer function (EOTF) is needed to express the brightness in the real world as the brightness corresponding to the storage method of the display device. For example, the brightness in the real world is 10000cd / m 2 , the display device uses 10 bits to store brightness information, so the maximum value that can be stored in this storage mode is 1023, so 10000cd / m 2 Represented as 1023.

[0122] Standard dynamic range (SDR) images, also known as low dynamic range, generally have a dynamic range of 1cd / m 2 to 100cd / m2 Images between and are compliant with the BT.709 or sRGB color gamut, and can use the gamma curve as the photoelectric conversion curve. SDR images correspond to HDR images. The 8-bit images we traditionally use, such as JPEG, can be considered SDR images. Before the advent of cameras capable of capturing HDR images, traditional cameras could only record the captured light information within a certain range by controlling the exposure value. Since the maximum brightness information of the display device cannot reach the brightness information of the real world, and we browse images through the display device, a photoelectric transfer function is required. Early display devices were cathode ray tube (CRT) displays, and their photoelectric transfer function was the gamma function.

[0123] With the upgrade of display devices, the illumination range of display devices continues to increase, and the illumination of existing HDR displays has reached 600cd / m 2 , the illumination of high-end HDR displays can reach 2000cd / m 2 Therefore, an improved photoelectric transfer function is needed to adapt to the upgrade of display devices. At present, the common photoelectric transfer functions include perception quantization (PQ) photoelectric transfer function, hybrid log-gamma (HLG) photoelectric transfer function, and scene luminance fidelity (SLF) photoelectric transfer function. The above three curves can refer to existing technologies and will not be described in detail in this application.

[0124] 3. Dynamic range mapping

[0125] Dynamic range mapping can be applied to the adaptation of the front-end HDR signal and the display device of the display end HDR. For example, the front-end captures 4000cd / m 2 The HDR display capability of the HDR display device on the display side is 500cd / m 2 , 4000cd / m 2 The light signal is mapped to 500cd / m 2 On the display device, it is a tone mapping (TM) process from high to low. Dynamic range mapping can also be applied to the adaptation of the front-end SDR signal and the display device of the display end HDR. For example, the front-end captures 100cd / m 2 The HDR display capability of the display device on the display end is 2000cd / m 2 , 100cd / m 2 The light signal is mapped to 2000cd / m 2On the display device, it is a TM process from low to high.

[0126] Currently, dynamic range mapping methods can be categorized as static and dynamic. Static mapping methods use a single piece of data to perform a comprehensive TM process based on the same video content or hard drive content. This means that the mapping curve is typically the same for all scenes. The advantage of this method is that the video requires less data and the processing flow is relatively simple. However, its disadvantage is that all scenes use the same mapping curve for TM, which can lead to information loss in some scenarios. For example, if the mapping curve prioritizes bright areas, details in extremely dark scenes may be lost or completely invisible, affecting the video's display quality. Dynamic mapping methods dynamically adjust the mapping curve for each scene or frame based on specific areas. This method has the advantage of enabling differentiated processing for different scenes or frames. However, its disadvantage is that each frame or scene requires relevant scene information, requiring a large amount of data to be carried by the video.

[0127] 4. Single-layer code stream

[0128] It can be an enhanced image code stream or a basic image code stream. Among them, the enhanced image code stream includes one or more enhanced images. The basic image code stream includes one or more basic images. Optionally, the HDR video (or HDR image) in the existing technology can be encoded and transmitted using H.265, etc., and the single-layer video (i.e., single-layer code stream) is transcoded during the transmission process. After decoding, the HDR video obtains frame-by-frame HDR pixel values ​​and HDR format identification and information. HDR video usually needs to be sent to the image processing module to complete image up and down sampling, enhancement and other operations. After processing, the HDR pixel value and HDR format identification and information are sent to the encoder for re-encoding.

[0129] 5. Dual-layer code stream

[0130] It includes basic image coding data (which may be referred to as first coding data in the embodiment of the present application) and enhanced image coding data (which may be referred to as second coding data in the embodiment of the present application). Among them, the basic image coding data is used to represent the enhanced image (which may also be referred to as the enhanced layer image, etc., which is not limited in the present application), and the enhanced image coding data is used to represent the basic image (which may also be referred to as the basic layer image, the basic image, the base layer image, etc., which is not limited in the present application). In the embodiment of the present application, the dual-layer code stream may include coding data of one or more basic images, and coding data of one or more enhanced images. In the process of processing images (including enhanced image data or basic image data) in the following embodiments, unless otherwise specified, it means that the same processing is performed on each image data. In the embodiment of the present application, the dual-layer code stream can be an image (or picture) code stream or a video code stream. That is, the enhanced image and the basic image in the dual-layer code stream can be pictures or videos, that is, the image display method in the embodiment of the present application can be applied to the processing of image code streams or to the processing of video code streams. Optionally, in some examples, the dual-layer code stream can also be understood as encapsulating the enhanced image code stream and the basic image code stream into one code stream, that is, after the receiving end or transcoding end receives the dual-layer code stream and processes the dual-layer code stream (which can be called decapsulation processing), the basic image code stream and the enhanced image code stream can be obtained. In other words, the various processing of the enhanced image described in the embodiments of the present application (including decoding, image processing and encoding, etc.) can be understood as processing the enhanced image stream (i.e., one or more enhanced images). The various processing of the basic image (including decoding, image processing and encoding, etc.) can be understood as processing the basic image stream (i.e., one or more basic images). This application does not limit this and will not be repeated below.

[0131] 6. Base image: describes an independent image data structure, including pixel and image-related metadata.

[0132] 7. Enhancement image: describes the enhanced image data structure, including pixel and image-related metadata.

[0133] 8. Synthetic image, also referred to as an extracted selectable image, selectable image, replacement image, or substitute image, is not limited in this application. Synthetic image is used to describe the image data structure obtained from the format specified in the embodiments of this application for display or subsequent processing, including pixel and image-related metadata.

[0134] 9. Metadata: data that describes image attributes and characteristics, as well as key information required during image processing.

[0135] In this application, the dual-layer code stream can be an image (or picture) code stream or a video code stream. In other words, the enhanced image data and basic image data in the dual-layer code stream can be either picture data or video data. That is, the image display method in this application can be applied to the processing of image code streams or video code streams.

[0136] FIG6 is a system block diagram of an exemplary receiving end. Referring to FIG6 , the receiving end includes but is not limited to at least one of the following: a decapsulation module, a decoding module, an enhanced image generation module, and a tone mapping module.

[0137] At the receiving end, after receiving the double-layer code stream, the decapsulation module decapsulates the code stream and extracts the basic image coding data, enhanced image coding data and metadata.

[0138] Next, the decoding module decodes the basic image coded data (e.g., in the embodiment of the present application, it may also be recorded as the first coded data), the enhanced image coded data (e.g., in the embodiment of the present application, it may also be recorded as the second coded data), and the metadata coded data, respectively, to obtain the basic image, the enhanced image, and the metadata. Specifically, the decoding module (which may also be referred to as the image decoding module, which is not limited in this application) is used to decode the basic image coded data to obtain the basic image; decode the enhanced image coded data to obtain the enhanced image; and decode the metadata coded data to obtain the metadata.

[0139] In this embodiment of the present application, each codestream includes at least one image (also referred to as image data). For example, after decoding the enhanced image coded data, at least one enhanced image (enhance1 ... enhanceN) can be obtained, and after decoding the base image coded data, at least one base image (base1 ... baseN) can be obtained. In this embodiment of the present application, the term "image data" is used to refer to the combination of at least one enhanced image and at least one base image, and this description will not be repeated below.

[0140] Optionally, the decoding module may decode the code stream using a decoder such as High Efficiency Video Coding (HEVC) or Joint Photographic Experts Group JPEG.

[0141] In one possible implementation, the image data (including the enhanced image and the base image) acquired by the decoding module can be image data in any color space format, such as RGB or YUV, and this application does not limit this. The color space format of the enhanced image and the base image can be the same or different, and this application does not limit this.

[0142] In another possible implementation, the bit width of the image data may be 8 bits, 10 bits, 12 bits, etc., which is not limited in this application.

[0143] In an embodiment of the present application, metadata can be embedded into a user-defined portion of the code stream during encoding. For example, the customized portion can be a Supplemental Enhancement Information (SEI) field in HEVC or Versatile Video Coding (VVC), or a customized Network Abstract Layer (NAL) unit, or other reserved fields. The customized portion can also be a JEIF-encapsulated APP extended information field or an MP4-encapsulated data segment, etc., which can be set according to actual needs and is not limited by this application.

[0144] Still referring to FIG. 6 , the enhanced image generation module (may also be referred to as a synthesis module) obtains a synthesized image based on the basic image and the enhanced image.

[0145] The image processing module (also referred to as a tone mapping module) may perform display adaptation processing on the synthesized image based on the metadata and display information of the electronic device, and output a target image.

[0146] The display module displays the target image on the display screen in response to the acquired target image, wherein the target image may also be referred to as a display image, etc., which is not limited in this application.

[0147] Optionally, the modules in FIG6 may be integrated on the same chip or on different chips. For example, the encapsulation module, decoding module, enhanced image synthesis module, and image processing module may be integrated on the same chip, and the display module may be integrated on another chip. The modules may communicate with each other via a bus, etc. The module configuration may be configured according to actual needs and is not limited in this application.

[0148] FIG7 is a flow chart of an image display method provided in an embodiment of the present application, which can be executed by the receiving electronic device described above (hereinafter referred to as the electronic device), such as the destination device, the terminal device, the terminal display device, the client device, the OTT device, etc. FIG7 is described as a series of steps or operations. It should be understood that the various processes in FIG7 can be executed in various orders and / or occur simultaneously, and are not limited to the execution order shown in FIG7. Specifically, it includes but is not limited to the following steps:

[0149] Step 701: Obtain a dual-layer code stream, where the dual-layer code stream includes first coded data, second coded data, and metadata. The first coded data is used to represent a basic image, and the second coded data is used to represent an enhanced image.

[0150] In the embodiment of the present application, the code stream may be a dual-layer code stream, including basic image coding data (for example, in the embodiment of the present application, it may be recorded as first coding data) and enhanced image coding data (for example, in the embodiment of the present application, it may be recorded as second coding data). The basic image coding data is used to represent the basic image, and the enhanced image coding data is used to represent the enhanced image. It can also be understood that the basic image coding data is generated after encoding the basic image, and the enhanced image coding data is generated after encoding the enhanced image.

[0151] Optionally, the basic image may include one or more basic images, and the enhanced image may include one or more enhanced images.

[0152] Optionally, the code stream includes metadata. In other embodiments, the metadata may also be a separate code stream, which is not limited in this application.

[0153] Exemplarily, the decapsulation module decapsulates the code stream and outputs an enhanced image code stream, a basic image code stream, and a metadata code stream.

[0154] In the embodiments of the present application, the dual-layer code stream can be an image (or picture) code stream or a video code stream. In other words, the enhanced image and the basic image in the dual-layer code stream can be either picture data or video data. In other words, the image display method in the embodiments of the present application can be applied to the processing of image code streams or video code streams.

[0155] In addition, the code stream may also include metadata. Optionally, the metadata may also be a separate code stream, which is not specifically limited in the embodiment of the present application.

[0156] In one possible implementation, the image (including the basic image and the enhanced image) obtained by parsing the code stream can be image data in any color space format such as RGB or YUV, and the embodiment of the present application does not specifically limit this.

[0157] In a possible implementation, the bit width of the image data may be 8 bits, 7 bits, or 12 bits, etc., which is not specifically limited in the embodiment of the present application.

[0158] In an embodiment of the present application, metadata can be embedded into a user-defined part of the code stream during encoding. For example, the user-defined part can be a Supplemental Enhancement Information (SEI) field in HEVC or Versatile Video Coding (VVC), or a customized Network Abstract Layer (NAL) unit, or other reserved fields. The user-defined part can also be an APP extension information field encapsulated by JEIF or a data segment encapsulated by MP4, etc. The aforementioned user-defined part can be set according to actual needs, and the embodiment of the present application does not specifically limit this.

[0159] In a possible implementation, the metadata includes but is not limited to at least one of the following information: source data format, region division information, region traversal order information, image features, and tone mapping information.

[0160] In an embodiment of the present application, the above-mentioned tone mapping information includes but is not limited to at least one of the following information: tone mapping information used for generating a synthetic image, tone mapping information used for displaying an image (also referred to as tone mapping information used for a target image), and adjustment parameters, etc.

[0161] Exemplarily, the tone mapping information used to generate the composite image includes, but is not limited to, base image tone mapping parameters and enhanced image tone mapping parameters. The base image tone mapping parameters are used to perform tone mapping on the base image, and the enhanced image tone mapping parameters are used to perform tone mapping on the enhanced image.

[0162] Exemplarily, the tone mapping information used to display an image includes, but is not limited to, at least one preset display information and tone mapping parameters corresponding to each preset display information. The concepts and types of preset display information can be found in the description of display information of electronic devices in the following embodiments and are not further elaborated here.

[0163] Optionally, the tone mapping information used in generating the composite image is used to participate in the composite image generation process. The tone mapping information used in generating the composite image can be understood as being used to remap the base image and / or the enhanced image to adjust the data volume of the base image and / or the enhanced image, for example, to reduce or increase the data volume. For example, at the transmitting end or the transcoding end, the image quality of the enhanced image and the base image may be reduced to reduce the data transmission bandwidth. At the receiving end, the receiving end may remap the base image and the enhanced image based on the first tone mapping information to obtain the image before the quality reduction, thereby improving the quality of the image displayed on the display end.

[0164] Optionally, tone mapping for displaying an image participates in the generation process of the display image, i.e., the process of tone mapping the composite image to generate the display image. This is primarily used to adapt the display image to the display capabilities of the display device. This can be understood as tone mapping a composite image that may not originally match the display capabilities of the display device to produce a display image that does match the display capabilities of the display device.

[0165] Optionally, the tone mapping parameters may be one or more of the three curves involved in the above embodiments, or other types of parameters, which are not limited in this application.

[0166] Step 702: Acquire a basic image and an enhanced image based on the first coded data and the second coded data.

[0167] Exemplarily, the electronic device may acquire a basic image based on the first encoding data, and acquire an enhanced image based on the second encoding data.

[0168] Specifically, the decoding module in the electronic device can decode the first coded data and the second coded data, respectively, to obtain a base image and an enhanced image. Exemplarily, the decoding module decodes the base image coded data (i.e., the first coded data) to obtain the base image. Furthermore, the decoding module decodes the enhanced image coded data (i.e., the second coded data) to obtain the enhanced image.

[0169] The decoding module decodes each code stream according to the standard corresponding to the code stream format, and outputs image data (including enhanced image and basic image) and metadata.

[0170] The decoder in the electronic device decodes the code stream according to the standard corresponding to the code stream format, and outputs an image (including a basic image and an enhanced image) and metadata.

[0171] Optionally, a decoder such as High Efficiency Video Coding (HEVC) or Joint Photographic Experts Group JPEG may be used to decode the code stream.

[0172] Step 703: Merge the basic image and the enhanced image to obtain a composite image.

[0173] Exemplarily, after the electronic device obtains the basic image and the enhanced image, it can generate the tone mapping information used based on the composite image in the metadata, perform tone mapping processing on the basic image and the enhanced image, and merge the processed basic image and enhanced image to obtain a composite image.

[0174] Specifically, the electronic device may optionally perform tone mapping on the base image to obtain a tone-mapped base image. Furthermore, the electronic device may optionally perform tone mapping on the enhanced image to obtain a tone-mapped enhanced image. The electronic device may synthesize the processed base image and the enhanced image to obtain an intermediate image. The electronic device may optionally further process the intermediate image based on adjustment parameters, etc., in the metadata to obtain a synthesized image.

[0175] Step 704: Perform tone mapping on the synthesized image based on first tone mapping information to obtain a first target image, where the first tone mapping information is determined based on the metadata and first display information of the electronic device.

[0176] Exemplarily, the electronic device places the acquired composite image in a corresponding cache area (which may be referred to as a composite image cache area). When the electronic device needs to process the composite image, it can be understood as processing the composite image in the cache area, or it can be understood as extracting the composite image in the cache area and performing corresponding processing, which is not limited in this application.

[0177] For example, the electronic device may perform tone mapping (also referred to as a tone mapping operation or tone mapping processing) on ​​the synthesized image based on the second tone mapping information to obtain a target image, which may also be referred to as a display image or an image to be displayed.

[0178] In an embodiment of the present application, the tone mapping information used for the target image is determined by the electronic device based on the display information of the electronic device and the tone mapping information used for the target image in the metadata (also referred to as the tone mapping information used for the display image).

[0179] Specifically, the metadata includes tone mapping information used for the target image. The tone mapping information used for the target image includes, but is not limited to, at least one tone mapping information and at least one preset display information corresponding to the at least one tone mapping information. The tone mapping information and the preset display information may optionally correspond one-to-one.

[0180] The display information of the electronic device includes but is not limited to: the display information of the display screen of the electronic device and the display environment information. Optionally, the display information of the display screen includes but is not limited to: at least one of: the brightness range of the display screen, the size of the display screen, etc. Optionally, the display environment information includes but is not limited to: at least one of: the brightness of the surrounding environment of the electronic device, the color temperature of the surrounding environment, etc. Optionally, the electronic device may include a module that records the display information of the display screen, and the electronic device may obtain the current display information of the display screen of the electronic device from the module. Optionally, the display information of the display screen may be variable, for example, the brightness range of the display screen may be changed in response to a received user operation, which is not limited in this application. Optionally, the electronic device may obtain environmental information such as ambient brightness and ambient color temperature through devices such as sensors. The specific acquisition method may refer to existing technologies, which is not limited in this application.

[0181] In the embodiments of the present application, the electronic device can obtain the first tone mapping information or the second tone mapping information involved in the embodiments below based on the two types of information (i.e., the tone mapping information used by the target image in the metadata and the display information of the electronic device). The embodiments of Figures 7 and 8 only use the method of obtaining the first tone mapping information as an example. The methods of obtaining the second tone mapping information and further tone mapping information are similar, and this application will not further illustrate each example one by one.

[0182] In embodiments of the present application, the first tone mapping information may include, but is not limited to, a tone mapping curve. Tone mapping curves can take various forms, such as sigmoid, cubic spline, gamma, and straight lines. Reference may be made to the description of the photoelectric transfer function above, and this embodiment of the present application does not specifically limit this. For example, the electronic device may calculate based on the curve to obtain a mapping relationship between normalized source data and normalized image data.

[0183] The embodiment of the present application does not limit the format of the metadata. The metadata may include histogram information and tone-mapping curve parameter information as in ST2094-40, or may include tone-mapping curve parameter information as in ST2094-10.

[0184] Optionally, the metadata may include information of multiple preset display devices, such as peak brightness, and a tone mapping curve for performing tone mapping on the peak brightness.

[0185] In a possible implementation, the electronic device may match the display information of the electronic device with at least one preset display information in the tone mapping information used for the target image in the metadata based on the display information of the electronic device.

[0186] Exemplarily, if the matching is successful, that is, at least one of the preset display information is the same as the display information on the electronic device, the electronic device may obtain the tone mapping curve corresponding to the preset display information that is successfully matched.

[0187] Optionally, if at least one preset display information does not contain display information that is identical to the display information of the electronic device. In one example, the electronic device can select a preset display information that is closest to the display information of the electronic device from at least one preset display information, and adjust the tone mapping information (for example, a tone mapping curve) corresponding to the preset display information based on the difference value between the preset display information and the display information of the electronic device. For example, if the current brightness of the electronic device is A, the brightness indicated in the preset display information closest to it in the metadata is B, and B is n times A. The electronic device can adjust the tone mapping curve corresponding to the preset display information based on the difference between brightness B and brightness A (i.e., n times), for example, multiplying the parameter value of the tone mapping curve by n (the specific processing method can be set according to actual needs, is only an illustrative example, and is not limited in this application.), to obtain the adjusted tone mapping curve (i.e., the first tone mapping information).

[0188] In another example, the electronic device may select multiple (e.g., two or more) of the most similar preset display information and, based on the difference between the electronic device's display information and the multiple preset display information, adjust the multiple tone mapping information corresponding to the multiple preset display information to obtain the first tone mapping information. Optionally, the adjustment method may employ a weighted average method, which can be set based on actual needs and is not limited in this application.

[0189] Of course, the method for obtaining the tone mapping information described in the embodiments of this application is merely an illustrative example and can be configured according to actual needs, and is not limited by this application. For example, if the difference between the preset display information and the display information of the electronic device is within a certain range (which can be configured according to actual needs), the tone mapping information can be directly used as the first tone mapping information without adjusting it.

[0190] Exemplarily, the electronic device may perform tone mapping on the synthesized image based on the acquired first tone mapping information to obtain the first target image.

[0191] Step 705: Display the first target image.

[0192] Exemplarily, the electronic device may display the first target image on a display screen. Alternatively, this step may be understood as the image processing module outputting the first target image to the display module, which then displays the first target image on the display screen. Of course, in some instances, the electronic device housing the image processing module and the display device housing the display module are separate, and this step may also be understood as the electronic device triggering the display device to display the first target image.

[0193] In one possible implementation, within the display duration corresponding to the first target image, the electronic device may change the display information of the electronic device in response to a received user operation. For example, in response to receiving a user operation, the display brightness of the electronic device is adjusted (i.e., the brightness peak is adjusted). The electronic device may optionally determine the second tone mapping information based on the metadata and the current display information of the electronic device (e.g., recorded as the second display information). The specific determination method may refer to the determination method of the first tone mapping information, which will not be repeated here. The electronic device may perform tone mapping on the composite image based on the second tone mapping information to obtain a second target image. And, the electronic device may display the second target image on the display screen. In this way, in this example, the second target image displayed by the electronic device is an image adapted to the current display capability of the electronic device to improve the display effect of the image. That is, as the display information of the electronic device is adjusted, the electronic device may re-determine the corresponding tone mapping information based on the dynamically changed display information, so that the generated target image can adapt to the dynamically changing scene of the display information.

[0194] FIG8 is a flow chart showing an exemplary method for displaying an image. The process in FIG7 will be described in detail below in conjunction with FIG8. Referring to FIG8, the process specifically includes but is not limited to the following steps:

[0195] S801a: The decoding module decodes the first coded data to obtain a basic image.

[0196] S801b: The decoding module decodes the second coded data to obtain an enhanced image.

[0197] Exemplarily, the decoding module obtains a double-layer code stream, which includes basic image coding data (for example, recorded as first coding data), enhanced image coding data (for example, recorded as second coding data), and metadata.

[0198] The decoding module decodes the basic image coded data to obtain a basic image.

[0199] The decoding module decodes the enhanced image coded data to obtain an enhanced image.

[0200] Optionally, S801a and S802b may be processed in parallel or sequentially, and this application does not limit the execution order.

[0201] S802: The decoding module outputs the enhanced image and the basic image to the enhanced image generation module.

[0202] Exemplarily, the decoding module outputs the enhanced image, the basic image and the metadata to the enhanced image generation module.

[0203] S803: The enhanced image generation module synthesizes the basic image and the enhanced image to obtain a synthesized image.

[0204] In embodiments of the present application, the enhanced image generation module may perform tone mapping on the base image and / or enhanced image based on the tone mapping information in the metadata (referring to the tone mapping information used for generating the composite image described above), and then merge the tone-mapped base image and enhanced image to obtain an intermediate image. Based on the intermediate image, a composite image is obtained.

[0205] The tone mapping process of the base image and the enhanced image, and the merging process of the base image and the enhanced image after tone mapping are described below.

[0206] 1. Tone Mapping of Basic Images

[0207] Tone mapping is performed on the basic image (denoted as Base) according to the basic image tone mapping information to obtain a tone-mapped basic image (denoted as baseAfter).

[0208] FIG9 is a schematic diagram of a tone mapping process according to an embodiment of the present application. Exemplarily, as shown in FIG9 , tone mapping is performed on a base image (e.g., base[i]) according to the base image tone mapping information to obtain a tone-mapped base image (e.g., baseAfter[i]), where i represents a pixel in the image.

[0209] For the convenience of description, the embodiment of the present application takes the tone mapping process of a basic image as an example to illustrate. The tone mapping processes of other basic images can refer to this process and will not be described one by one.

[0210] In the embodiment of the present application, the tone mapping process of the basic image may include but is not limited to the following methods:

[0211] Method 1: baseAfter[i]=base[i]

[0212] In this method, the tone-mapped basic image is the basic image, which can also be understood as not undergoing tone mapping processing.

[0213] Method 2: baseAfter[i]=base[i]*THH+(A-base[i])*THL

[0214] Here, THH is the upper limit of the basic image data, and THL is the lower limit of the basic image data. It can be understood that THH and THL indicate the data range of the basic image, and A is the maximum value stored in the basic image. It can be understood that method 2 normalizes the basic image data to the range indicated by THH and THL. When the basic image data is normalized between 0 (corresponding to THL) and 1.0 (corresponding to THH), A is 1.0.

[0215] Optionally, THH and THL are tone mapping parameters, included in the tone mapping information in the metadata. Optionally, A is a pre-set value. That is, the transmitter encodes based on this A value, and the receiver also performs tone mapping based on this A value. The specific value can be set according to actual needs and is not specifically limited in this embodiment of the application.

[0216] Optionally, THH and THL may be pre-set, that is, the agreed values ​​between the transmitting end and the receiving end. They may be set according to actual needs and are not specifically limited in the present embodiment.

[0217] Method 3: baseAfter[i]=base[i]+THL

[0218] Here, THL is the lower limit of the basic image data. It can be understood that THL is used to indicate the data range of the basic image, that is, it must not be lower than THL. It can be understood that method three is to normalize the basic image data so that its values ​​are normalized to a value range not lower than THL.

[0219] Optionally, THL is a tone mapping parameter, which is included in the tone mapping information of the metadata.

[0220] Optionally, THL may also be pre-set, that is, a value agreed upon between the sending end and the receiving end.

[0221] Method 4: baseAfter[i]=THH+A-base[i]

[0222] Here, THH is the upper limit of the basic image data. It can be understood that THH indicates the data range of the basic image, that is, not exceeding (or exceeding) THH; A is the maximum value stored in the basic image. It can be understood that method four normalizes the basic image data to a value not exceeding the range indicated by THH. When the basic image data is normalized to between 0 and 1.0, the value of A is 1.0.

[0223] Optionally, THH is a tone mapping parameter, which is included in the tone mapping information of the metadata.

[0224] Optionally, THH may also be pre-set, that is, an agreed value between the sending end and the receiving end.

[0225] Method 5: baseAfter[i]=TMB(base[i])

[0226] Wherein, TMB() is tone mapping information. Optionally, TMB() can be a global and / or local tone mapping function. In this manner, global tone mapping processing or local tone mapping processing can be performed on the basic image based on TMB().

[0227] Optionally, the tone mapping information of the metadata may also include some parameters in the TMB() function for expressing a mapping relationship, and the decoding function may determine the corresponding TMB() function based on these parameters.

[0228] Optionally, TMB() may also be preset, that is, a value agreed upon between the sending end and the receiving end.

[0229] Method 6: baseAfter[i]=TMB()*base[i]

[0230] Wherein, TMB() is tone mapping information. Optionally, TMB() can be a global and / or local tone mapping function. In this manner, global tone mapping processing or local tone mapping processing can be performed on the base based on the TMB() function.

[0231] In this way, tone mapping processing based on the tone mapping function can make the reconstructed HDR image data closer to the original HDR image data when it is subsequently reconstructed based on baseAfter, making the data size of the enhanced image data in the second bitstream (the concept of which can be found below) smaller.

[0232] Optionally, the tone mapping information of the metadata may also include parameters in the TMB() function for expressing a mapping relationship, and the corresponding TMB() function may be determined based on these parameters.

[0233] Optionally, TMB() may also be preset, that is, a value agreed upon between the sending end and the receiving end.

[0234] Method 7: baseAfter[i]=TMB(TMB1(base[i]))

[0235] Among them, TMB() and TMB1() are both tone mapping information. Optionally, TMB() can be a global and / or local tone mapping function. Optionally, TMB1() can be in the form of linear, spline, segmented curve, etc. The role of TMB1() can be understood as being used to indicate that the basic image base[i] is transformed into a specified range. In other words, the tone mapping information in Methods 2 to 4 (such as THL or THH) can also be represented by TMB1(), and it can be understood that the corresponding TMB1() can be obtained based on the tone mapping information (such as THH, etc.). Of course, TMB1() in Method 7 (and Method 8) can also be represented by any of Methods 2 to 4, and the embodiments of this application do not make specific limitations.

[0236] Optionally, the tone mapping information of the metadata may also include parameters in the TMB() and TMB1() functions for expressing a mapping relationship, and the corresponding TMB() and TMB1() functions may be determined based on these parameters.

[0237] Optionally, TMB() and / or TMB1() may also be preset, that is, the agreed value between the sending end and the receiving end.

[0238] Method 8: baseAfter[i]=TMB()*TMB1(base[i])

[0239] Among them, TMB() and TMB1() are both tone mapping information. Optionally, TMB() can be a global and / or local tone mapping function. Optionally, TMB1() can be in the form of linear, spline, segmented curve, etc. The role of TMB1() can be understood as being used to indicate that the basic image base[i] is transformed into a specified range. In other words, the tone mapping information in Methods 2 to 4 (such as THL or THH) can also be represented by TMB1(), and it can be understood that the corresponding TMB1() can be obtained based on the tone mapping information (such as THH, etc.). Of course, TMB1() in Method 7 (and Method 8) can also be represented by any of Methods 2 to 4, and the embodiments of this application do not make specific limitations.

[0240] Optionally, the tone mapping information of the metadata may also include parameters in the TMB() and TMB1() functions for expressing a mapping relationship, and the corresponding TMB() and TMB1() functions may be determined based on these parameters.

[0241] Optionally, TMB() and / or TMB1() may also be preset, that is, the agreed value between the sending end and the receiving end.

[0242] Method 9:

[0243] baseAfter[i]=TMB()*TMB1(baseMid1[i]); or, baseAfter[i]=TMB(TMB1(baseMid1[i]));

[0244] Among them, baseMid1[i]=F[base[i]] or baseMid1[i]=∑F[n]*base[i+n], baseAfter2[i]=base[i]-baseMid1[i]

[0245] Among them, the tone mapping information includes but is not limited to: TMB(), TMB1() and F[]. Optionally, TMB() can be a global and / or local tone mapping function. Optionally, TMB1() can be in the form of linear, spline, segmented curve, etc. The role of TMB1() can be understood as being used to indicate that the basic image base[i] is transformed into a specified range. That is to say, the tone mapping information in Methods 2 to 4 (such as THL or THH) can also be represented by TMB1(), and it can be understood that the corresponding TMB1() can be obtained based on the tone mapping information (such as THH, etc.). Of course, TMB1() in Method 7 (and Method 8) can also be represented by any of Methods 2 to 4, and the embodiments of the present application do not make specific limitations.

[0246] Optionally, F[] is a spatial filter parameter or other image smoothing function. baseMid1[i] and baseAfter2[i] can be obtained based on F[]. There are various filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not specifically limited in this embodiment of the application.

[0247] Optionally, the tone mapping information of the metadata may also include parameters in the TMB(), TMB1() and F[] functions for expressing mapping relationships, and the corresponding TMB(), TMB1() and F[] functions may be determined based on these parameters.

[0248] Optionally, at least one of TMB(), TMB1() and F[] may also be preset, that is, a value agreed upon between the transmitting end and the receiving end.

[0249] Method 10: baseAfter[i]=F[base[i]], or baseAfter[i]=∑F[n]*base[i+n] baseAfter2[i]=base[i]-baseMid1[i]

[0250] Where F[] is a spatial filter parameter or other image smoothing function. baseMid1[i] and baseAfter2[i] can be obtained based on F[]. Various filtering methods exist, such as bilateral filtering, interpolation filtering, and the like, which are not specifically limited in this embodiment.

[0251] Optionally, the tone mapping information of the metadata may also include parameters in the F[] function for expressing a mapping relationship, and the corresponding F[] function may be determined based on these parameters.

[0252] Optionally, F[] may be pre-set, that is, a value agreed upon between the sender and the receiver.

[0253] 2. Enhanced Image Tone Mapping

[0254] Tone mapping is performed on the enhanced image (denoted as enhance) according to the enhanced image tone mapping information to obtain an enhanced image after tone mapping (denoted as enhanceAfter).

[0255] Exemplarily, as shown in FIG9 , tone mapping is performed on the enhanced image (eg, enhance[i]) according to the enhanced image tone mapping information to obtain a tone-mapped enhanced image (eg, enhanceAfter[i]), where i represents a pixel in the image.

[0256] For the convenience of description, the embodiment of the present application takes the tone mapping process of an enhanced image as an example to illustrate. The tone mapping processes of other enhanced images can refer to this process and will not be described one by one.

[0257] In the embodiment of the present application, the tone mapping process of enhancing the image may include but is not limited to the following methods:

[0258] Method 1: enhanceAfter[i]=enhance[i]

[0259] In this method, the tone-mapped enhanced image is the enhanced image, which can also be understood as not having been subjected to tone mapping processing.

[0260] Method 2: enhanceAfter[i]=enhance[i]*THH+(A-enhance[i])*THL

[0261] Here, THH is the upper limit of the enhanced image data, and THL is the lower limit of the enhanced image data. It can be understood that THH and THL indicate the data range of the enhanced image, and A is the maximum value stored for the enhanced image. It can be understood that method 2 normalizes the enhanced image data to the range indicated by THH and THL. When the enhanced image data is normalized between 0 (corresponding to THL) and 1.0 (corresponding to THH), A is 1.0.

[0262] Optionally, THH and THL are tone mapping parameters, included in the tone mapping information in the metadata. Optionally, A is a pre-set value. That is, the transmitter encodes based on this A value, and the receiver also performs tone mapping based on this A value. The specific value can be set according to actual needs and is not specifically limited in this embodiment of the application.

[0263] Optionally, THH and THL may be pre-set, that is, the agreed values ​​between the transmitting end and the receiving end, and may be set according to actual needs, and are not specifically limited in the embodiments of the present application.

[0264] Method 3: enhanceAfter[i]=enhance[i]+THL

[0265] Here, THL is the lower limit of the enhanced image data. It can be understood that THL is used to indicate the data range of the enhanced image, i.e., it must not be lower than THL. It can be understood that method three normalizes the enhanced image data to a value not lower than the range indicated by THL.

[0266] Optionally, THL is a tone mapping parameter, which is included in the tone mapping information of the metadata.

[0267] Optionally, THL may also be pre-set, that is, a value agreed upon between the sending end and the receiving end.

[0268] Method 4: enhanceAfter[i]=THH+A-enhance[i]

[0269] Here, THH is the upper limit of the enhanced image data. It can be understood that THH indicates the data range of the enhanced image, that is, not exceeding (or exceeding) THH; A is the maximum value stored for the enhanced image. It can be understood that method four normalizes the enhanced image data to a value not exceeding the range indicated by THH. When the enhanced image data is normalized to between 0 and 1.0, the value of A is 1.0.

[0270] Optionally, THH is a tone mapping parameter, which is included in the tone mapping information of the metadata.

[0271] Optionally, THH may also be pre-set, that is, an agreed value between the sending end and the receiving end.

[0272] Method 5: enhanceAfter[i]=TME(enhance[i])

[0273] Wherein, TME() is tone mapping information. Optionally, TME() can be a global and / or local tone mapping function. In this manner, global tone mapping processing or local tone mapping processing can be performed on the enhanced image based on TME().

[0274] Optionally, the tone mapping information of the metadata may also include parameters in the TME() function for expressing a mapping relationship, and the corresponding TME() function may be determined based on these parameters.

[0275] Optionally, TME() may also be preset, that is, the agreed value between the sending end and the receiving end.

[0276] 3. Merge the tone-mapped base image and the enhanced image to obtain an intermediate image.

[0277] According to the tone-mapped base image (denoted as baseAfter) and the tone-mapped enhanced image (denoted as enhanceAfter), an intermediate image (denoted as recHDR) is obtained.

[0278] For example, as shown in FIG9 , intermediate image data (e.g., recHDR[i]) is obtained based on the tone-mapped base image (e.g., baseAfter[i]) and the tone-mapped enhanced image (enhanceAfter[i]). This can also be understood as merging the tone-mapped base image and the tone-mapped enhanced image to obtain the corresponding intermediate image.

[0279] For the convenience of description, the embodiment of the present application takes the generation process of an intermediate image as an example to illustrate. The generation process of other intermediate images can refer to this process and will not be described one by one.

[0280] In the embodiment of the present application, the generation process of the intermediate image may include but is not limited to the following methods:

[0281] Method 1: recHDR[i]=baseAfter[i]*enhanceAfter[i]

[0282] Method 2: recHDR[i]=baseAfter[i]+enhanceAfter[i]

[0283] Method 3: recHDR[i]=f1(baseAfter[i])+f2(enhanceAfter[i])

[0284] Method 4: recHDR[i]=f1(baseAfter[i])*f2(enhanceAfter[i])

[0285] Wherein, f1() and f2() are conversion functions in the numerical domain. Exemplarily, the tone-mapped base image and / or the tone-mapped enhanced image can be transformed into the same numerical domain based on the f1() and f2() functions and then merged (including addition or multiplication).

[0286] It should be noted that the merging methods involved in the embodiments of the present application are only illustrative examples. In other embodiments, other feasible merging methods can also be adopted, and the embodiments of the present application do not make specific limitations.

[0287] Optionally, the tone mapping information of the metadata may include information for indicating any merging scheme, and a corresponding scheme is selected based on the indication of the metadata.

[0288] Optionally, the merging method may also be pre-set, that is, an agreed value between the sending end and the receiving end.

[0289] 4) Based on the intermediate image, obtain the composite image (denoted as HDRPic)

[0290] Specifically, as described above, the tone mapping information of the metadata may further include adjustment parameters. The enhanced image generation module may perform numerical adjustments on at least one intermediate image (recHDR_1 . . . recHDR_N) based on the adjustment parameters to obtain a composite image.

[0291] The adjustment method is as follows:

[0292] Method 1: HDRPic[i] = recHDR[i]

[0293] In this method, it can also be understood that no numerical adjustment is made, that is, the numerical adjustment amplitude is 0. Of course, it can also be understood that the intermediate image is the composite image, that is, the enhanced image generation module merges the processed basic image and the enhanced image to obtain the composite image.

[0294] Method 2: HDRPic[i]=recHDR[i]+X

[0295] Where X is the adjustment parameter.

[0296] Optionally, X may be an adjustment parameter in the tone mapping information of the metadata, or may be preset by the transcoding end, which is not limited in this application.

[0297] Method 3: HDRPic[i]=g(recHDR[i])

[0298] Here, g() is an inverse normalization function, a mapping function, a tone mapping function, or an inverse mapping function.

[0299] Optionally, g() can be an adjustment parameter in the tone mapping information of the metadata, or it can be preset by the transcoder. This application does not limit this. The decoding function can determine the corresponding g() function based on these parameters. Optionally, g() can be a global or local tone mapping function. In this method, the enhanced image generation module can perform global tone mapping or local tone mapping on the enhanced image based on the g() function.

[0300] It should be noted that the numerical adjustment step is optional. If the transmitting end performs numerical adjustment during the encoding process, the receiving end will perform numerical adjustment accordingly. Conversely, if the transmitting end does not perform numerical adjustment during the encoding process, the receiving end does not need to perform numerical adjustment. Of course, it can also be understood that the numerical adjustment is performed, but the numerical adjustment amplitude is 0 (i.e., method 1 of numerical adjustment).

[0301] In one possible implementation, as described above, the decoded enhanced image and base image may be one or more. In the case where there are multiple enhanced images and / or base images, the above processing may be performed on each of the multiple images to obtain recHDR[i]. The specific processing is as follows:

[0302] Method 1: recHDR[i]=A*g(baseAfter1[i]*f1(enhanceAfter1[i]))+B*g(baseAfter2[i]*f2(enhanceAfter2[i]))+C* g(baseAfter3[i]*f3(enhanceAfter3[i]))…

[0303] Among them, f1(), f2() and f3() are conversion functions. The above functions may be included in the tone mapping information in the metadata, or they may be preset (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or it may be preset, and the embodiments of the present application do not make specific limitations. Optionally, A, B, C... etc. are preset constants, or are included in the tone mapping information of the metadata.

[0304] Method 2: recHDR[i]=A*g(baseAfter1[i]*f1(enhanceAfter1[i]))+B*g(baseAfter2[i])+C*g(baseAfter3[i])…

[0305] Among them, f1() is a conversion function. The above function may be included in the tone mapping information in the metadata, or it may be preset (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or it may be preset, and the embodiments of the present application do not make specific limitations. Optionally, A, B, C... etc. are preset constants, or are included in the tone mapping information of the metadata.

[0306] Exemplarily, this method can be used to process a scene in which a code stream includes one enhanced image and multiple basic images.

[0307] Method three: recHDR[i]=A*g(baseAfter1[i]+f1(enhanceAfter1[i]))+B*g(baseAfter2[i]+f2(enhanceAfter2[i]))+C* g(baseAfter3[i]+f3(enhanceAfter3[i]))…

[0308] Among them, f1(), f2() and f3() are conversion functions. The above functions may be included in the tone mapping information in the metadata, or they may be preset (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or it may be preset, and the embodiments of the present application do not make specific limitations. Optionally, A, B, C... etc. are preset constants, or are included in the tone mapping information of the metadata.

[0309] Method 4: recHDR[i]=A*g(baseAfter1[i]+f1(enhanceAfter1[i]))+B*g(baseAfter2[i])+C*g(baseAfter3[i])…

[0310] Among them, f1() is a conversion function. The above function may be included in the tone mapping information in the metadata, or it may be preset (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or it may be preset, and the embodiments of the present application do not make specific limitations. Optionally, A, B, C... etc. are preset constants, or are included in the tone mapping information of the metadata.

[0311] Exemplarily, this method can be used to process a scene in which a code stream includes one enhanced image and multiple basic images.

[0312] Method 5: recHDR[i]=A*baseAfter1[i]+A1*f1(enhanceAfter1[i])+baseAfter1[i]+B*baseAfter2[i]+B1* f1(enhanceAfter2[i])+C*baseAfter3[i]+C1*f1(enhanceAfter3[i])…

[0313] Among them, f1(), f2() and f3() are conversion functions. The above functions may be included in the tone mapping information in the metadata, or they may be preset (i.e., pre-agreed). f() is used to perform numerical domain conversion on the enhanced image and / or the basic image. The detailed description can be referred to above and will not be repeated here. g() is an inverse normalization function. Optionally, g() may be an adjustment parameter in the tone mapping information of the metadata, or it may be preset, and the embodiments of the present application do not make specific limitations. Optionally, A, B, C... etc. are preset constants, or are included in the tone mapping information of the metadata.

[0314] It should be noted that the embodiments of the present application do not limit baseAfter[i] and enhanceAfter[i] to any domain, and can be a linear domain, PQ domain, LOG domain, etc. The embodiments of the present application also do not limit the color space of baseAfter[i] and enhanceAfter[i], and can be YUV, RGB, Lab, HSV, etc. color space.

[0315] It should be further explained that after the composite image is obtained, other image processing may be further performed on the composite image, which is not specifically limited in the embodiment of the present application.

[0316] S804: The enhanced image generation module outputs the synthesized image to the image processing module.

[0317] Exemplarily, the enhanced image generation module outputs the synthesized image to the image processing module.

[0318] S805: The image processing module obtains first display information of the electronic device.

[0319] Exemplarily, the display information of the electronic device includes but is not limited to: display information of the display screen and display environment information. In an embodiment of the present application, the electronic device can obtain the display information of the display screen and the display environment information.

[0320] Optionally, the display information of the display screen includes but is not limited to at least one of the following: peak brightness, and / or darkest brightness, size, etc.

[0321] Optionally, the display environment information includes but is not limited to: ambient brightness, ambient color temperature, etc.

[0322] It should be noted that this application does not limit the display information obtained by the display screen to be updated in real time or factory preset values. This application also does not limit the type of display information, and it can also include display-related parameters such as ambient light.

[0323] For other descriptions, please refer to the above and will not be repeated here.

[0324] S806: The image processing module determines first tone mapping information based on the metadata and the first display information.

[0325] The specific processing method can be found in Figure 7 and will not be described here in detail.

[0326] S807: The image processing module performs tone mapping on the synthesized image based on the first tone mapping information to obtain a first target image.

[0327] The image processing module performs tone mapping on the synthesized image according to the adjusted tone mapping curve to obtain a first target image. Exemplarily, the tone mapping may include the following process:

[0328] 1) Obtain the gain table lut according to the adjusted tone mapping curve;

[0329] 2) For each pixel (R, G, B), obtain its brightness value lum and obtain the gain coefficient ratio from the gain table lut. For example:

[0330] Lum=max(R,G,B), or Lum=a*R+b*G+c*B, where a, b, c are weighting coefficients.

[0331] ratio=lut[Lum]

[0332] It should be noted that the embodiment of the present application does not limit the calculation method of lum, and other brightness calculation methods such as L in Lab or I in ICtCp can be used.

[0333] 3) Adjust the brightness of each pixel (R, G, B). For example,

[0334] RTM=R*Ratio, GTM=G*Ratio, BTM=B*Ratio;

[0335] or

[0336] [YUV] = M * [RGB], where M is selected according to BT.2020 or BT709, YTM = Y * ratio, and [RGB] = M - 1 * [YUV];

[0337] 4) Adjust the saturation of each pixel (R, G, B). For example:

[0338] Obtaining ratioc for color adjustment according to Lum and ratio, which may be ratioc=(ratio)N;

[0339] [YUV]=M*[RGB], M is selected according to BT.2020 or BT709, UTM=U*ratioc, VTM=V*ratioc.

[0340] It should be noted that the embodiment of the present application does not limit the saturation (color) adjustment algorithm, and it can be performed in color domains such as RGB, IctCp, HSV, and Lab.

[0341] It should be noted that color gamut mapping can be added after processing to convert from the current color gamut to the target color gamut. The current color gamut and the target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0342] The above tone mapping method is only an illustrative example, and may include more or fewer processing methods, which is not limited in this application.

[0343] S808: The image processing module outputs the first target image to the display module.

[0344] Exemplarily, the image processing module executes a display sending process, that is, outputs the first target image to the display module.

[0345] S809: The display module displays the first target image.

[0346] Exemplarily, the display module receives the first target image and displays the first target image on the display screen.

[0347] Optionally, after the electronic device detects a change in display information, the method may further include the following steps. The change in display information of the electronic device may be caused by the electronic device adjusting display parameters of the electronic device in response to a received user operation and determining the change in display information. Alternatively, if the change is due to environmental information, the electronic device may also detect in real time whether the environmental information in the display information has changed based on parameters uploaded by sensors or the like.

[0348] S810: The image processing module obtains second display information of the electronic device.

[0349] Exemplarily, the electronic device obtains current second display information, i.e., the adjusted display information. The second display information is different from the first display information. The difference may be a difference in display information on the display screen and / or in environmental information, which is not limited in this application. For other undescribed portions, please refer to the above and will not be repeated here.

[0350] S811: The image processing module determines second tone mapping information based on the metadata and the second display information.

[0351] S812: The image processing module performs tone mapping on the synthesized image based on the second tone mapping information to obtain a second target image.

[0352] S813: The image processing module outputs the second target image to the display module.

[0353] S814: The display module displays the second target image.

[0354] For the parts not described in the above process, please refer to the above text and will not be repeated here.

[0355] In summary, in the embodiments of the present application, by synthesizing the basic image and the enhanced image after decoding, a single synthesized image is obtained to adapt to a single buffer, solving the inapplicable situation existing in the processing method of the single-layer bitstream. In existing operating systems such as Android, the decoding module (whether hardware decoding or software decoding), the enhanced image generation module (invoking the GPU or software API for image processing), the image processing module, and the display module (invoking the display hardware) are usually independent of each other, and may even be cross-vendor modules. Usually, only one image buffer and additional metadata are passed between the above modules. After decoding, the double-layer structure bitstream usually contains two image buffers, namely the basic image and the enhanced image. In the present application, the decoding, processing, and display of the double-layer bitstream are separated from each other. After decoding, the two obtained images are merged into a single synthesized image, so that the buffers of the two images are merged into one buffer. In this way, from decoding to obtaining the image and then to display, the API interfaces between the existing modules can be reused. When displaying, tone mapping can be performed according to the display-related information transmitted by the metadata, and finally the correct processing effect can be obtained, avoiding the trouble of re-designing the application program because it cannot use the original interface.

[0356] FIG. 10 is a schematic flowchart of another exemplary image display method. Referring to FIG. 10, it specifically includes but is not limited to the following steps:

[0357] In the embodiments of the present application, after any step before executing S703 in FIG. 7, the electronic device can obtain the allowable range of tone distortion of the electronic device.

[0358] Exemplarily, after the image is processed, adapted, and adjusted, the tone of the pixels often changes, that is, compared with before processing, adaptation, and adjustment, the tone of some pixels has changed, and the tone distortion degree is used to reflect the degree of this change. Exemplarily, the pixel distortion degree can be represented by a percentage, that is, A / B*100%, where A is the number of pixels with tone changes in the image, and B is the total number of pixels in the image.

[0359] The allowable range of the tone distortion degree: reflects the tolerance degree of the user or the receiving end (or display device) to tone distortion. Exemplarily, the chromaticity distortion degree can be a preset range, such as "0% < X < 15%", "5% < X < 15%", where X is the tone distortion degree. The former indicates that the user / terminal device allows a tone distortion degree not greater than 15%, and the latter indicates that the user / terminal device allows a tone distortion degree between 5% and 15%.

[0360] In a possible implementation, after the receiving end obtains the allowable range of hue distortion, it can determine the display mode of the dual-layer image according to the allowable range of hue distortion. Optionally, the receiving end can set multiple preset ranges. In the embodiments of this application, the receiving end sets a first preset range and a second preset range. Optionally, the first preset range can be "5% < X < 1115%", and the second preset range can be "0% < X ≤ 5%", which can be set according to actual needs and are not limited in this application.

[0361] In one example, if the allowable range of hue distortion is within the first preset range, the processing flow of S703 and subsequent steps in FIG. 7 can be continued.

[0362] In another example, if the allowable range of hue distortion is within the second preset range, the processing flow in FIG. 10 can be executed.

[0363] S1001, The enhanced image generation module obtains the display information of the electronic device.

[0364] The execution process can be referred to the above, and will not be elaborated here.

[0365] S1002, The enhanced image generation module performs image processing on the enhanced image based on the display information to obtain a second enhanced image.

[0366] Exemplarily, the enhanced image generation module can perform image processing on the enhanced image based on the obtained display information. Optionally, the image processing is to perform numerical scaling on the enhanced image based on the display information. For example, taking the brightness value corresponding to the enhanced image as A and the brightness value corresponding to the display information as B, the electronic device can magnify or reduce the value of the enhanced image based on the difference value (such as a multiple) between the brightness value A and the brightness value B to obtain a second enhanced image.

[0367] S1003, Based on the basic image and the second enhanced image, a composite image is obtained.

[0368] Exemplarily, the enhanced image generation module can obtain a composite image based on the basic image and the second enhanced image. The specific processing can be referred to the above, and will not be elaborated here.

[0369] S1004, The enhanced image generation module outputs the composite image to the display module.

[0370] S1005, The display module displays the composite image.

[0371] That is to say, in this example, the composite image is used as the target image and is displayed on the display screen.

[0372] In the scenario shown in Figure 10, if the electronic device adjusts its display information in response to a received user operation, the enhanced image generation module needs to obtain the updated display information and, based on the new display information, process the enhanced image to obtain a third enhanced image. A new composite image is then generated based on the base image and the third enhanced image. The display module then displays the new composite image.

[0373] The embodiment of the present application further provides an image display device 1100 , see FIG11 , which is an example diagram of an image display device provided in the embodiment of the present application, and is used to implement the method in the embodiment of the present application.

[0374] As shown in FIG11 , the image display device 1100 may include a processor 1101 for executing programs or instructions stored in a memory 1102 . When the programs or instructions stored in the memory 1102 are executed, the processor 1101 is configured to execute the image display method described in the above embodiments.

[0375] Optionally, the image processing apparatus 1100 may further include a communication interface 1103. FIG11 uses a dotted line to indicate that the communication interface 1103 is optional for the image processing apparatus 1100.

[0376] Among them, the number of processors 1101, memories 1102 and communication interfaces 1103 does not constitute a limitation on the embodiments of the present application. During specific implementation, they can be arbitrarily configured according to business requirements.

[0377] Optionally, the memory 1102 is located outside the image processing device 1100 .

[0378] Optionally, the image display device 1100 includes the memory 1102, which is connected to the at least one processor 1101 and stores instructions executable by the at least one processor 1101. FIG11 uses dotted lines to indicate that the memory 1102 is optional for the image processing device 1100.

[0379] The processor 1101 and the memory 1102 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0380] The specific connection medium between the processor 1101, memory 1102, and communication interface 1103 is not limited in the embodiments of the present application. In Figure 11, the processor 1101, memory 1102, and communication interface 1103 are connected via a bus 1104. The bus is represented by a bold line in Figure 11. The connection between other components is only for schematic illustration and is not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bold line is used in Figure 11, but this does not mean that there is only one bus or one type of bus.

[0381] It should be understood that the processors mentioned in the embodiments of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that is implemented by reading software code stored in a memory.

[0382] Exemplarily, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0383] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0384] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0385] It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0386] An embodiment of the present application also provides a computer-readable storage medium, including a program or instruction, which, when executed on a computer, causes the method in FIG. 7 , FIG. 8 , or FIG. 10 to be executed.

[0387] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0388] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.

[0389] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0390] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0391] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is intended to include such modifications and variations.

Claims

1. An image display method, characterized in that: include: Acquire a double-layer code stream, the double-layer code stream comprising first coded data, second coded data, and metadata, the first coded data being used to represent a basic image, and the second coded data being used to represent an enhanced image; Based on the first coded data and the second coded data, acquiring the basic image and the enhanced image; Merging the basic image and the enhanced image to obtain a composite image; Performing tone mapping on the synthesized image based on first tone mapping information to obtain a first target image, wherein the first tone mapping information is determined based on the metadata and first display information of the electronic device; The first target image is displayed.

2. The method according to claim 1, characterized in that The display information of the electronic device includes at least one of the following: Display information and display environment information of the display screen.

3. The method according to claim 2, characterized in that The display information of the display screen includes at least one of the following: The brightness range of the display screen and the size of the display screen.

4. The method according to claim 2, characterized in that: The display environment information includes at least one of the following: The brightness of the environment surrounding the electronic device and the color temperature of the environment surrounding the electronic device.

5. The method according to claim 1, characterized in that The metadata includes at least one tone mapping information and at least one preset display information corresponding to the at least one tone mapping information. Before tone mapping the synthesized image based on the first tone mapping information, the method further includes: Based on the display information of the electronic device, determining target preset display information matching the display information of the electronic device; Based on the difference value between the target preset display information and the display information of the electronic device, the tone mapping information corresponding to the target preset display information is adjusted to obtain the first tone mapping information.

6. The method according to claim 1, characterized in that The metadata includes at least one tone mapping information and at least one preset display information corresponding to the at least one tone mapping information. Before tone mapping the synthesized image based on the first tone mapping information, the method further includes: Based on the difference value between the display information of the electronic device and the at least one preset display information, the at least one tone mapping information is adjusted to obtain the first tone mapping information.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Performing tone mapping on the synthesized image based on second tone mapping information to obtain a second target image; wherein the second tone mapping information is determined based on the metadata and second display information of the electronic device; The second target image is displayed.

8. The method according to claim 1, characterized in that Before merging the basic image and the enhanced image, the method further includes: Determines the allowable range of tonal distortion.

9. The method according to claim 8, characterized in that The merging of the basic image and the enhanced image comprises: When the allowable range of the hue distortion is within a preset range, the basic image and the enhanced image are merged to obtain the composite image.

10. An image display device, characterized in that: include: An acquisition module is used to acquire a double-layer code stream, wherein the double-layer code stream includes first coded data, second coded data and metadata. The first coded data is used to represent the basic image, and the second coded data is used to represent the enhanced image; A decoding module, configured to obtain the basic image and the enhanced image based on the first coded data and the second coded data; A synthesis module, used for merging the basic image and the enhanced image to obtain a synthesized image; a tone mapping module, configured to perform tone mapping on the synthesized image based on first tone mapping information to obtain a first target image, wherein the first tone mapping information is determined based on the metadata and first display information of the electronic device; A display module is used to display the first target image.

11. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 9.

Images