Image processing method and device, and storage medium

Abstract

This application discloses an image processing method and apparatus, as well as a storage medium, in order to support various image layering methods, provide a flexible image fusion manner, and help improve image processing effect, and relates to the field of video image processing. The method includes: parsing an original bitstream of a target image to obtain a target image; parsing a first layered bitstream of the target image to obtain a first layered image of the target image; determining a second layered image of the target image according to the target image and the first layered image, where the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is the detail layer image and the second layered image is a base layer image; performing tone mapping on the base layer image to obtain a mapping result of the base layer image; and fusing the mapping result of the base layer image and the detail layer image.

Description

[Technical field]

[0001] The present application relates to the field of image processing, and in particular to image processing methods and apparatus, and storage media. [Background technology]

[0002] At present, image processing techniques are widely applied in various scenarios to improve image quality. Usually, the conversion between high dynamic range (HDR) images and low dynamic range (LDR) images is implemented by using image layering and tone mapping methods.

[0003] An image mainly includes luminance information and edge (contour) information. Luminance information determines the whole view of the image. Edge (contour) information reflects the essential attributes of objects in the image and does not change significantly with luminance. Therefore, for the two kinds of information contained in the image, the image can be decomposed into a base layer image and a detail layer image, and this technique is called image layering technique.

[0004] In the field of imaging, dynamic range (DR) describes the ratio of the maximum grayscale value to the minimum grayscale value within the displayable range of an image. Natural dynamic ranges are large. For example, the luminance of a starry night scene is about 0.01 candela per square meter (cd / m 2 As another example, the luminance of the sun is 1,000,000,000 cd / m 2 The natural dynamic range is 10 13The dynamic range of the image captured by the camera can reach the order of magnitude of the pixel count. However, the displayable dynamic range of the image captured by the camera is small. In order to ensure the imaging effect, the dynamic range needs to be reduced, and the characteristics such as contrast and detail of the HDR image need to be preserved as much as possible. This process is called tone mapping (TM).

[0005] In the prior art, the method of converting HDR images to low dynamic range (LDR) is not diversified. Usually, pre-processing is performed on the HDR image, including converting the color space of the HDR image to CIELab, performing normalization of the HDR image, etc. Image layering is performed on the pre-processed HDR image by using a fast bilateral filtering method, tone mapping and saturation adjustment are performed on the base layer image, and processing is performed on the detail layer image. Finally, the processed base layer image and the processed detail layer image are combined to form a compressed LDR image. In this method, since the above fixed image processing method is used in the same device, different HDR images cannot be flexibly adjusted to achieve optimal image processing effects. Summary of the Invention

[0006] The embodiments of the present application provide an image processing method and apparatus, and a storage medium, to help support various image layering methods, provide a flexible image fusion scheme, and improve image processing effects.

[0007] To achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0008] According to a first aspect, there is provided an image processing method, comprising: parsing an original bitstream of an image to be processed to obtain an image to be processed; parsing a first layered bitstream of the image to be processed to obtain a first layered image of the image to be processed; determining a second layered image of the image to be processed based on the image to be processed and the first layered image, wherein the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is the detail layer image and the second layered image is a base layer image; performing tone mapping on the base layer image to obtain a mapping result of the base layer image; and fusing the mapping result of the base layer image and the detail layer image.

[0009] In the current HDR image-based processing method, a fixed image layering method, a tone mapping method, an image fusion method, etc. are used, so that the optimal effect cannot be achieved in image processing. In this case, according to the image processing method provided in the present application, the image processing device parses the original bit stream and the bit stream of the first layered image to obtain the image to be processed, obtains the second layered image according to the image to be processed and the first layered image, and performs image processing on the first layered image and the second layered image separately. The processing methods for different layered images can be flexibly adjusted, and image fusion is performed on the flexibly processed first layered image and the second layered image, which helps to improve the image processing effect. In other words, in the present application, the image layering task in the process of compressing HDR images can be placed on other devices other than the image processing device. However, in the prior art, the image layering and the process of processing and fusing the image obtained after layering are performed on the same device. In the process of implementing multiple image processing steps, the same device necessarily imposes restrictions. The method of the present application helps to achieve the effect of not restricting the specific method of image layering. Other devices send the bitstream obtained after image layering to the image processing device, and the image processing device performs flexible processing and fusion based on the layered image to help improve the image processing effect of compressing the HDR image.

[0010] In a possible implementation, the first layered bitstream is a downsampled bitstream, and parsing the first layered bitstream of the image to be processed to obtain a first layered image of the image to be processed includes parsing the first layered bitstream of the image to be processed to obtain a first initial layered image of the image to be processed, and performing upsampling on the first initial layered image to obtain the first layered image.

[0011] In this possible implementation, the image processing device has an upsampling function, and the transmitted bitstream is the bitstream obtained after downsampling, i.e. the data amount of the transmitted bitstream is small, which helps to reduce the resources occupied in the bitstream transmission process, thereby reducing the bandwidth usage.

[0012] In a possible implementation, the first layered image and the second layered image are single channel images, or alternatively, the first layered image and the second layered image are multi-channel images.

[0013] In this possible implementation, the specific format of the image is not limited, which helps to increase the flexibility and feasibility of the solution.

[0014] In a possible implementation, fusing the mapping result of the base layer image with the detail layer image includes obtaining a processed detail layer image based on the detail layer image and a weighting factor, where the weighting factor represents the overlay intensity of the detail layer image, and fusing the mapping result of the base layer image with the processed detail layer image.

[0015] In this possible implementation, a specific implementation of image fusion is provided, in which the processing degree of the detail layer image is flexibly changed, so as to provide personalized processing for different images, and improve the image processing effect.

[0016] In a possible implementation, prior to parsing the original bitstream of the image to be processed and parsing the first layered bitstream of the image to be processed, the method further includes receiving a dual bitstream, the dual bitstream including the original bitstream and the first layered bitstream.

[0017] In this possible implementation, the image processing device performs a synchronization process based on the image to be processed and the layered image, and performs fusion on the processed images.

[0018] According to a second aspect, an image processing device is provided, the image processing device including a functional unit configured to perform any of the methods provided in the first aspect, and the operation performed by each function is implemented by hardware or by the hardware executing corresponding software. For example, the image processing device may include a parsing unit, a determining unit, a tone mapping unit, and a fusing unit. The parsing unit is configured to parse an original bitstream of a processing target image to obtain a processing target image, the parsing unit is further configured to parse a first layered bitstream of the processing target image to obtain a first layered image of the processing target image, the determining unit is configured to determine a second layered image of the processing target image based on the processing target image and the first layered image, the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is the detail layer image and the second layered image is a base layer image, the tone mapping unit is configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image, and the fusing unit is configured to fuse the mapping result of the base layer image and the detail layer image.

[0019] For technical effects achieved by implementation of the second aspect, please refer to the technical effects achieved by the corresponding implementation of the first aspect.

[0020] According to a third aspect, there is provided an image processing method, the method comprising: receiving an original bitstream of a to-be-processed image and a first layered bitstream of the to-be-processed image, discarding the first layered bitstream, parsing the original bitstream to obtain the to-be-processed image, performing image layering on the to-be-processed image to obtain a base layer image and a detail layer image, performing tone mapping on the base layer image to obtain a mapping result of the base layer image, and fusing the mapping result of the base layer image and the detail layer image.

[0021] It is believed that the existing partial image processing device does not have dual bitstream processing capability, so by using the image processing method provided in this application, after the original bitstream and the first layered bitstream are received, the first layered bitstream can be discarded, the original bitstream is parsed to obtain a target image, and image layering, tone mapping and fusion are performed on the target image according to the existing image processing method to obtain a fused image. The above method improves the backward compatibility and robustness of the image processing device.

[0022] According to a fourth aspect, there is provided an image processing device, the image processing device including functional units configured to perform any of the methods provided in the third aspect, and the operations performed by each function are implemented by hardware or by the hardware executing corresponding software. For example, the image processing device may include a receiving unit, a processing unit, a parsing unit, a tone mapping unit, and a fusion unit. The receiving unit is configured to receive an original bitstream of a processing target image and a first layered bitstream of the processing target image, the processing unit is configured to discard the first layered bitstream, the parsing unit is configured to parse the original bitstream to obtain the processing target image, the processing unit is further configured to perform image layering on the processing target image to obtain a base layer image and a detail layer image, the tone mapping unit is configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image, and the fusion unit is configured to fuse the mapping result of the base layer image and the detail layer image.

[0023] For technical effects achieved by implementation of the fourth aspect, please refer to the technical effects achieved by the corresponding implementation of the third aspect.

[0024] According to a fifth aspect, there is provided an image processing apparatus comprising a processor and an interface circuit, the processor configured to receive or transmit data by using the interface circuit, the processor configured to invoke program instructions stored in a memory such that the image processing apparatus performs any of the methods provided in the first or third aspect.

[0025] According to a sixth aspect, there is provided an image processing apparatus comprising a processor and a memory, the processor being connected to the memory, the memory being configured to store computer executable instructions, the processor executing the computer executable instructions stored in the memory to perform any of the methods provided in the first or third aspect.

[0026] According to a seventh aspect, there is provided a chip. The chip includes a processor and an interface circuit. The interface circuit is configured to receive code instructions and communicate the code instructions to the processor. The processor is configured to execute the code instructions to perform any of the methods provided in the first or third aspect.

[0027] According to an eighth aspect, a computer-readable storage medium is provided, comprising computer-executable instructions which, when executed on a computer, enable the computer to perform any of the methods provided in the first or third aspect.

[0028] According to a ninth aspect, there is provided a computer program product, comprising computer executable instructions which, when executed on a computer, enable the computer to perform any of the methods provided in the first or third aspect.

[0029] For the technical effects brought about by any of the implementations in the fifth to ninth aspects, please refer to the technical effects brought about by the corresponding implementations in the first or third aspects, and the details will not be described again here. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a diagram of the structure of an image processing device according to an embodiment of the present application. [Diagram 2] FIG. 1 is a diagram of the structure of a video coding device according to an embodiment of the present application. [Diagram 3]1 is a schematic flow chart of an image capture device and an image processing apparatus according to an embodiment of the present application. [Figure 4] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Diagram 5] FIG. 2 is a diagram of a local tone mapping method according to an embodiment of the present application. [Figure 6a] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Figure 6b] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Figure 7] 1 is a schematic flow chart of a multi-channel image processing method according to an embodiment of the present application; [Figure 8] 1 is a schematic flow chart of a downsampling image processing method according to an embodiment of the present application; [Figure 9] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Figure 10] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Figure 11] 1 is a diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present application. [Figure 12] 1 is a diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] In the present specification, unless otherwise specified, " / " means "or". For example, A / B may represent A or B. The term "and / or" in this specification merely indicates an association relationship between related objects, and indicates that three relationships may exist. For example, A and / or B may represent the following three cases: only A exists, both A and B exist, and only B exists. In addition, "at least one" means one or more, and "multiple" means two or more. Terms such as "first" and "second" do not limit the quantity and execution order, and terms such as "first" and "second" do not indicate clear differences.

[0032] It should be noted that, in this application, terms such as "example" or "for example" are used to denote serving as an example, illustration, or description. Any embodiment or design scheme described in this application as an "example" or "for example" should not be described as preferred or having more advantages over other embodiments or design schemes. Indeed, use of terms such as "example", "for example", and the like is intended to present the relevant concepts in a concrete manner.

[0033] The embodiment of the present application provides an image processing method that can be applied to an image processing device. The specific form of the image processing device is not limited in this embodiment of the present application. For example, the image processing device may specifically be a terminal device. The terminal device may be called a terminal, user equipment (UE), terminal device, video coding device, access terminal, subscriber unit, subscriber station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication device, user agent, user equipment, etc. The terminal device may specifically be a mobile phone, a camera, a display, an augmented reality (AR) device, a virtual reality (VR) device, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc.

[0034] In the present application, the structure of an image processing device may be shown in Fig. 1. Fig. 1 is a diagram of a hardware structure of an image processing device 10 according to an embodiment of the present application. The image processing device 10 may be configured to implement the functions of the image processing device.

[0035] 1 may include a processor 101, a memory 102, a communication interface 103, and a bus 104. The processor 101, the memory 102, and the communication interface 103 may be connected via the bus 104.

[0036] Processor 101 is the control center of image processing device 10 and may be a general purpose central processing unit (CPU), other general purpose processor, etc. The general purpose processor may be a microprocessor, any conventional processor, etc.

[0037] For example, processor 101 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG.

[0038] Memory 102 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying the expected program code in the form of instructions or data structures and accessible by a computer.

[0039] In a possible implementation, the memory 102 may be independent of the processor 101. The memory 102 may be connected to the processor 101 via a bus 104 and is configured to store data, instructions or program codes. When retrieving and executing the instructions or program codes stored in the memory 102, the processor 101 can implement the image processing method provided in the embodiments of the present application.

[0040] In other possible implementations, the memory 102 may alternatively be integrated with the processor 101 .

[0041] The communication interface 103 is configured to connect the image processing device 10 to other devices by using a communication network. The communication network may be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 103 may include a receiving unit configured to receive data and a transmitting unit configured to transmit data.

[0042] The bus 104 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus may be classified as an address bus, a data bus, a control bus, etc. For simplicity of illustration, only one thick line is used in FIG. 1 for illustration purposes, but it does not represent that there is only one bus or one type of bus.

[0043] It should be noted that the structure shown in Fig. 1 does not constitute a limitation on the image processing device 10. In addition to the components shown in Fig. 1, the image processing device 10 may include more or fewer components than those shown, may combine some of the components, or may have a different component arrangement.

[0044] For example, when the image processing device is a video coding device, the processor further includes a coding module. FIG. 2 is a diagram of a hardware structure of a video coding device 200 according to an embodiment of the present application. The video coding device may be a decoding device or an encoding device. The video coding device 200 is suitable for implementing the disclosed examples / embodiments as described herein. The video coding device 200 includes a downlink port 220, an uplink port 250, and / or a transceiver unit (Tx / Rx) 210. The transceiver unit 210 includes a transmitting device and / or a receiver configured to perform uplink and / or downlink data communication over a network. The video coding device 200 further includes a processor 230 and a memory 240. Video coding device 200 may further include electrical, optical-to-electrical (OE), electrical-to-optical (EO), and / or wireless communication components coupled to uplink port 250 and / or downlink port 220 for communication of data over an electrical, optical, or wireless communication network. Video coding device 200 may further include input and / or output (input / output, I / O) devices 260 for communication of data with a user. I / O devices 260 may include output devices such as a display for displaying video data and speakers for outputting audio data. I / O devices 260 may also include input devices such as a keyboard, mouse, trackball, and / or corresponding interfaces for interacting with such output devices.

[0045] The processor 230 is implemented by hardware and software. The processor 230 may be implemented as one or more CPU chips, one or more cores (e.g., implemented as a multi-core processor), one or more field-programmable gate arrays (FPGAs), one or more application-specific integrated circuits (ASICs), and one or more digital signal processors (DSPs). The processor 230 communicates with the downlink port 220, the Tx / Rx 210, the uplink port 250, and the memory 240. The processor 230 includes a coding module 232. The coding module 232 performs the steps of parsing the bitstream in the following disclosed embodiments, such as steps S201 and S202 shown in FIG. 4. The coding module 232 may also perform any other method / mechanism described herein. In addition, the processor 230 may further include other modules configured to process the image parsed by the coding module 232, the multiple modules working together to complete the image processing methods provided herein.

[0046] It should be noted that a video coding device is configured to implement a video encoding / decoding function, which may in fact alternatively be implemented by using other types of devices, such as a decoding circuit.

[0047] At present, common methods for image layering include image layering by using filters or image layering by using artificial intelligence (AI) methods. Image layering is performed by using filters, and the base layer image after image layering is obtained by extracting low frequency information from the image, where the low frequency information is the area with low change in grayscale value, and the detail layer image after image layering is obtained by extracting high frequency information from the image, where the high frequency information is the area with high change in grayscale value. It can be seen that the area with low change in grayscale value presents the gradient state of the image, and the area with high change in grayscale value presents the obvious difference of the image, such as the contour of the object. When filters are used to perform image layering, hardware deployment needs to be performed in the image processing device. The cost and complexity of implementation are high, and maintenance and upgrade are inconvenient. When AI methods are used to perform image layering, high computing power of the image processing device is required, and a network with good layering effect but large scale cannot be deployed in this method. In conclusion, in this application, the image layering task is placed on other devices, and the specific method for image layering is not limited, so that the image processing device can flexibly set the image processing method, and different HDR images can be flexibly adjusted during image processing to achieve the optimal effect. As shown in FIG. 3, the capture device obtains an HDR image, performs pre-production on the HDR image, such as performing image pre-processing, and then performs image layering. The image layering method is not limited in this application. A part of the layered image and the original image are compressed into a bitstream and transmitted to the image processing device provided in this application. The image processing device parses the bitstream and performs image processing on the layered image separately, and then obtains an LDR image by image fusion to compress the HDR image. A flexible image processing and fusion method is used to improve the compression effect of the HDR image. The following describes the technical solutions provided in the embodiments of this application with reference to the accompanying drawings.

[0048] 4 is a flow chart of an image processing method according to the present application. The method includes the following steps:

[0049] S201: The image processing device parses the original bitstream of the target image to obtain the target image.

[0050] The original bitstream is a data stream sent to the image processing device after the target image is compressed, and the image processing device obtains the target image by parsing the original bitstream.

[0051] S202: The image processing device parses a first layered bitstream of the image to be processed to obtain a first layered image of the image to be processed.

[0052] The first layered bit stream is a data stream in which an image at an arbitrary layer after image layering is performed on the image to be processed is compressed and transmitted to the image processing device. The image processing device obtains the first layered image by parsing the first layered bit stream.

[0053] The first layered image is a base layer image or a detail layer image.

[0054] Optionally, the first layered image is a single-channel image or a multi-channel image. The number of image channels is determined based on the image color mode. For example, when the image color mode is a red, green, blue (RGB) mode, it indicates that the image is formed by parameters of three channels: R, G and B, and the image can be called a three-channel image. As another example, when the image mode is a grayscale mode, the image is formed by parameters of one channel, and the image can be called a single-channel image. In addition, the multi-channel image can also include a dual-channel image, a four-channel image, and so on.

[0055] It should be noted that the image to be processed may be a single-channel image or a multi-channel image. When the image to be processed is a single-channel image, the first layered image is a single-channel image. When the image to be processed is a multi-channel image, the first layered image is a single-channel image or a multi-channel image. Specifically, it is assumed that the number of channels of the image to be processed is N, the number of channels of the first layered image is M, N is an integer greater than 1, and 1≦M≦N.

[0056] Optionally, the first layered bitstream is a downsampled bitstream. Step S202 specifically includes: the image processing device parses the first layered bitstream of the target image to obtain a first initial layered image of the target image. The image processing device performs upsampling on the first initial layered image to obtain the first layered image.

[0057] The first initial layered image is a downsampled image, and the image on which upsampling is performed is an upsampled image. The downsampled image is an image output after zooming out, and the upsampled image is an image output after zooming in.

[0058] If the first layered bitstream is a downsampled bitstream, the image processing device parses the first layered bitstream to obtain a first initial layered image, and performs upsampling on the first initial layered image, in other words, zooms in the first initial layered image to obtain the first layered image, the resolution of the first layered image being the same as that of the image to be processed, whereby a second layered image is subsequently obtained.

[0059] Optionally, before step S201 and step S202 are performed, the method further includes step S200: an image processing device receiving an original bitstream and a first layered bitstream.

[0060] The original bitstream and the first layered bitstream are a single bitstream or a dual bitstream transmitted to the image processing device. The encapsulation format of the single bitstream is different from that of the dual bitstream in the bitstream. When the original bitstream and the first layered bitstream are transmitted to the image processing device in the form of a dual bitstream, the image processing device receives the dual bitstream, and the dual bitstream includes the original bitstream of the to-be-processed image and the first layered bitstream of the to-be-processed image. The image processing device may synchronously parse the original bitstream and the first layered bitstream to obtain the to-be-processed image and the first layered image and perform an image processing process. When the original bitstream and the first layered bitstream are transmitted to the image processing device in the form of a single bitstream, the bitstream may further include synchronization information, which is used by the image processing device to determine corresponding positions of pixels in the to-be-processed image and the first layered image to perform an image processing process, such as obtaining another layered image or performing image fusion.

[0061] It can be understood that when the original bitstream and the first layered bitstream are transmitted to the image processing device in the form of a dual bitstream, the image processing device performs synchronization processing based on the image to be processed and the layered image, and performs fusion on the processed image, so as to shorten the image processing time.

[0062] It should be noted that the present application does not strictly require the method of parsing the original bitstream and the first layered bitstream. When the original bitstream and the first layered bitstream are transmitted to the image processing device in the form of a dual bitstream, the image processing device may receive the dual bitstream and parse the dual bitstream synchronously, or may parse the original bitstream and the first layered bitstream separately according to sequence.

[0063] S203: The image processing device determines a second layered image of the processing target image based on the processing target image and the first layered image, the first layered image being a base layer image and the second layered image being a detail layer image, or the first layered image being the detail layer image and the second layered image being the base layer image.

[0064] It can be understood that image layering is performed on the to-be-processed image to obtain a base layer image and a detail layer image. Thus, the to-be-processed image includes a base layer image and a detail layer image. The image processing device obtains a second layered image by calculating a difference between the to-be-processed image and the first layered image.

[0065] Optionally, the image processing device calculates pixel differences between pixels of the image to be processed and the first layered image, and uses the image formed by the pixel differences as the second layered image.

[0066] Optionally, the image processing device calculates a difference between the original bitstream and the first layered bitstream, and parses the calculated difference bitstream to obtain a second layered image.

[0067] It can be understood that the first layered image and the second layered image have the same number of channels. If the first layered image is a single channel image, the second layered image is a single channel image. If the first layered image is a multi-channel image, the second layered image is a multi-channel image with the same number of channels.

[0068] Currently, for HDR images, tone mapping is mainly performed on the luminance information of an image, and then the edge information of the image is fused to obtain a fused LDR image. The contrast of an image reflects the degree of brightness in an image, and the degree of brightness is determined based on the luminance information. To restore the contrast in an HDR image, tone mapping is generally performed on the luminance information of an image. To restore the detail features in an HDR image, image processing is generally performed on the edge information of an image. A similar effect to that in an HDR image is achieved by fusion of the luminance information and the edge information, so that an image with a similar effect is presented to a user on different hardware devices.

[0069] Specifically, the image processing device calculates luminance information Y source and luminance information Y in the first layered image 1 Extract the following formula: Y 2 =Y source -Y 1 The luminance information Y of the second layered image is calculated according to 2 Get the.

[0070] Here, the first layered image is a base layer image, and the second layered image is a detail layer image. 1 is Y base may be expressed as Y 2 is Y detail Alternatively, the first layered image is a detail layer image, and the second layered image is a base layer image, and in this case, Y 1 is Ydetail It may be expressed as Y 2 is Y base It may be expressed as:

[0071] It can be understood that the luminance information is in units of pixels, and the luminance information of each pixel is represented by using a pixel value. In the calculation process, the pixel value of each pixel is used for calculation.

[0072] S204: The image processing device performs tone mapping on the base layer image to obtain a mapping result of the base layer image.

[0073] The specific method of tone mapping is not limited in this application. Tone mapping includes global tone mapping (GTM), local tone mapping (LTM), and AI tone mapping based on deep neural network. GTM means that pixel values ​​of all pixels in a base layer image are mapped by using the same mapping function. LTM means that pixel values ​​of pixels in different areas of a base layer image are mapped by using different mapping functions. LTM may divide an image into multiple image blocks, and the same or different mapping functions are used for the multiple image blocks. For example, as shown in FIG. 5, an image is divided into multiple image blocks, and the same mapping function is used for the image blocks filled with the same format. For example, a shaded image block means that mapping function 2 is used for pixel values ​​in the image block, and a black filled image block indicates that mapping function 3 is used for pixel values ​​in the image block.

[0074] It should be noted that the image processing device may use different tone mapping methods for different target images, and the image processing device may be selected according to the tone mapping methods supported by the device.

[0075] Optionally, if the image processing device supports at least two tone mapping methods, the image processing device performs image processing by using multiple tone mapping methods on the same processing target image, compares the processing effects of different tone mapping methods, and selects the processing result with the optimal effect.

[0076] S205: The image processing device fuses the mapping result of the base layer image with the detail layer image.

[0077] Specifically, the image processing device overlaps the mapping result of the base layer image and the detail layer image to obtain a fusion image.

[0078] Optionally, the image processing device obtains a processed detail layer image based on the detail layer image and a weighting factor, the weighting factor representing a blending strength of the detail layer image, the blending strength indicating a proportion of the processed detail layer image in the fusion image, and the image processing device blends the mapping result of the base layer image with the processed detail layer image.

[0079] It can be understood that the current detail layer image contains high-frequency information in the image. After the detail layer image is processed based on different weighting coefficients, the high-frequency information in the image is actually enhanced or weakened, so that the mapping results of the processed detail layer image and the base layer image have different effects during image fusion. In other words, the different weighting coefficients help to enhance or weaken the high-frequency information in the image, so as to indirectly affect the proportion of high-frequency information in the fusion image in the image fusion process.

[0080] Specifically, the processed detail layer image is Y' detail where the weighting factor is weight and Y' detail is calculated in the following way: Y' detail =Y detail×weight

[0081] where the weights are constants greater than 0. For example: weight=α

[0082] where α>0. The image processor blends pixel values ​​of the mapping result of the base layer image with pixel values ​​in the detail layer image with the same overlapping strength, i.e. weight=((Y tm / Y base )+α) β

[0083] Here, α>0 and β>0. The image processing device separately determines a weighting factor for fusion with the detail layer image based on the pixel value of the mapping result of the base layer image. Specifically, the image processing device determines a ratio of each pixel value of the mapping result of the base layer image to the pixel value of the corresponding position in the base layer image, and determines a weighting factor of the pixel based on the ratio.

[0084] After the image processing device determines the processed detail layer image, the processed detail layer image is converted into the mapping result Y tm and the fused image Y is expressed by the following formula: Y=Y tm +Y' detail It is calculated by using

[0085] Alternatively, Y is: Y=Y tm +Y detail ×weight It can be expressed as:

[0086] Through steps S201 to S205, the image processing device performs more flexible processing on the image. The image processing device obtains the processing target image and some layered images, performs various processing on the layered images, and adaptively adjusts the image fusion effect, which is helpful to improve the image processing effect and the user experience.

[0087] Steps S201 to S205 can be specifically implemented by using the flowchart shown in FIG. 6a or FIG. 6b.

[0088] If the first layered bitstream is a bitstream of a base layer image, please refer to the image processing process shown in Figure 6a. If the first layered bitstream is a bitstream of a detail layer image, please refer to the image processing process shown in Figure 6b.

[0089] Optionally, if the image obtained by the image processing device parsing the first layered bitstream is a multi-channel image (e.g., a three-channel image), please refer to the image processing process shown in Figure 7.

[0090] Where: weight1=α1, weight2=α2, and weight3=α3, where α1, α2, and α3 are all constants greater than zero, or, weight1=((C tm1 / C base1 )+α1) β1 weight2=((C tm2 / C base2 )+α2) β2 , and weight3=((C tm3 / C base2 )+α3) β3 and where α1, α2, α3, β1, β2, and β3 are all constants greater than zero.

[0091] Accordingly, the image processing device comprises: C1=C tm1 +C' detail1 , C2=C tm2 +C' detail2 , and C3=C tm3 +C' detail3 Calculate the pixel values ​​of the fused image as follows:

[0092] Where: C' detail =C detail ×weight

[0093] So the pixel values ​​of the fused image are: C1=C tm1 +C detail1 ×weight1, C2=C tm2 +C detail2 ×weight2, C3=C tm3 +C detail3 ×weight3 It can also be expressed as:

[0094] It can be understood that when the image to be processed and the first layered image are both multi-channel images, image processing can be performed independently on each channel, and the fusion image obtained by the image processing is also a multi-channel image.

[0095] It should be noted that in Fig. 7, an example is used in which the first layered image is a base layer image, but the first layered image may alternatively be a detail layer image. In the image processing process shown in Fig. 6b, the relevant specific parameters are the same as those in Fig. 7. Details are not described again.

[0096] Optionally, if the image obtained by the image processing device parsing the first layered bitstream is a downsampled first layered image, the following uses the downsampled base layer image as an example. As shown in Figure 8, the image processing device performs upsampling on the downsampled base layer image to obtain a base layer image, and then performs a subsequent image processing process.

[0097] When the bitstream is a downsampled first layered bitstream and transmitted to an image processing device, the transmission bandwidth is saved and the image processing efficiency is improved.

[0098] Optionally, other image processing and fusion methods after layering are provided. As shown in FIG. 9, tone mapping and image fusion are performed synchronously, and a fused image is output. It is believed that the overlapping position between the detail layer image and the base layer image obtained by tone mapping can be more flexibly selected, which can extend the device path design of the image processing device, improve the feasibility of achieving the optimal effect of tone mapping, and facilitate implementation by ensuring compatibility with existing image processing devices.

[0099] Optionally, if it is considered that the existing partial image processing device does not have dual bitstream processing capability, as shown in Fig. 10, after receiving the original bitstream and the first layered bitstream, the image processing device discards the first layered bitstream and obtains a target image by paring the original bitstream, and performs image layering, tone mapping and fusion on the target image according to the existing image processing method to obtain a fusion image. The above method improves the backward compatibility and robustness of the device of the image processing device.

[0100] Optionally, the bitstream may be non-gamma encoded; common formats include perceptual quantization (PQ) and hybrid log-gamma (HLG).

[0101] The above mainly describes the solutions in the embodiments of the present application from the perspective of a method. To implement the above functions, the image processing device includes at least one corresponding hardware structure and corresponding software module for performing each function. Those skilled in the art should easily realize that the present application can be implemented by hardware or a combination of hardware and computer software in combination with the example units and algorithm steps described in the embodiments disclosed herein. Whether the functions are implemented by hardware or hardware driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but it should not be considered that the implementation goes beyond the scope of the present application.

[0102] In the embodiment of the present application, the division into functional units may be performed in the image processing device based on the above example of the method. For example, the functional units may be divided based on the corresponding functions, or two or more functions may be integrated into one processing unit. The integrated units may be implemented in the form of hardware or in the form of software functional units. It should be noted that in this embodiment of the present application, the division into units is merely an example and is merely a logical division of functions. In actual implementation, other division methods may be used.

[0103] For example, FIG. 11 is a possible diagram of the structure of an image processing device (denoted as image processing device 90) in the above embodiment. The image processing device 90 includes a parsing unit 901, a determining unit 902, a tone mapping unit 903, and a fusing unit 904. The parsing unit 901 is configured to parse an original bitstream of a target image to obtain a target image, which is, for example, step S201 shown in FIG. 4. The parsing unit 901 is further configured to parse a first layered bitstream of the target image to obtain a first layered image of the target image, which is, for example, step S202 shown in FIG. 4. The determining unit 902 is configured to determine a second layered image of the target image based on the target image and the first layered image, where the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is a detail layer image and the second layered image is a base layer image, which is, for example, step S203 shown in FIG. 4. The tone mapping unit 903 is configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image, which is for example step S204 shown in Fig. 4. The fusion unit 904 is configured to fuse the mapping result of the base layer image and the detail layer image, which is for example step S205 shown in Fig. 4.

[0104] Optionally, the first layered bitstream is a downsampled bitstream, and the parsing unit 901 is particularly configured to parse the first layered bitstream of the image to be processed to obtain a first initial layered image of the image to be processed, and to perform upsampling on the first initial layered image to obtain the first layered image.

[0105] Optionally, the first layered image and the second layered image are single channel images, or the first layered image and the second layered image are multi-channel images.

[0106] Optionally, the fusion unit 904 is specifically configured to obtain a processed detail layer image based on the detail layer image and a weighting coefficient representing the overlay strength of the detail layer image, and fuse the mapping result of the base layer image with the processed detail layer image.

[0107] Optionally, the image processing device 90 further includes a receiving unit 905, where the receiving unit 905 is configured to receive a dual bitstream, where the dual bitstream includes an original bitstream and a first layered bitstream.

[0108] Optionally, the image processing device 90 further includes a storage unit 906. The storage unit 906 is configured to store computer-executable instructions, and other units in the image processing device can perform corresponding operations based on the computer-executable instructions stored in the storage unit 906.

[0109] For specific descriptions of any of the above aspects, please refer to the above method embodiments. Details will not be described again here. Furthermore, for any descriptions of the image processing device 90 given above and descriptions of the advantageous effects, please refer to the above corresponding method embodiments. Details will not be described again here.

[0110] In an example, referring to Fig. 1, some or all of the functions performed by the parsing unit 901, the determining unit 902, the tone mapping unit 903, the fusing unit 904, the receiving unit 905, and the storage unit 906 in the image processing device 90 may be implemented by the processor 101 in Fig. 1 by executing program codes in the memory 102 in Fig. 1. Alternatively, the receiving unit 905 may be implemented by using a receiving unit in the communication interface 103 in Fig. 1.

[0111] For example, Fig. 12 is a possible diagram of the structure of an image processing device (denoted as image processing device 100) in the above embodiment. The image processing device 100 includes: a receiving unit 1001 configured to receive an original bitstream of a processing target image and a first layered bitstream of the processing target image, a processing unit 1002 configured to discard the first layered bitstream, a parsing unit 1003 configured to parse the original bitstream to obtain a processing target image, the processing unit 1002 further configured to perform image layering on the processing target image to obtain a base layer image and a detail layer image, a tone mapping unit 1004 configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image, and a fusion unit 1005 configured to fuse the mapping result of the base layer image and the detail layer image.

[0112] Optionally, the image processing device 100 further includes a storage unit 1006. The storage unit 1006 is configured to store computer-executable instructions, and other units in the image processing device can perform corresponding operations based on the computer-executable instructions stored in the storage unit 1006.

[0113] For specific description of any of the above aspects, please refer to the above method embodiments. Details will not be described again here. Furthermore, for any description of the image processing device 100 given above and the description of the advantageous effects, please refer to the above corresponding method embodiments. Details will not be described again here.

[0114] In an example, referring to Fig. 1, some or all of the functions performed by the receiving unit 1001, the processing unit 1002, the parsing unit 1003, the tone mapping unit 1004, the fusion unit 1005, and the storage unit 1006 in the image processing device 100 may be implemented by the processor 101 in Fig. 1 by executing program codes in the memory 102 in Fig. 1. Alternatively, the receiving unit 1001 may be implemented by using a receiving unit in the communication interface 103 in Fig. 1.

[0115] An embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium storing a computer program, which, when executed on a computer, causes the computer to perform the method performed by the image processing device provided above.

[0116] For the description of the related contents and the description of the advantageous effects in any computer-readable storage medium provided above, please refer to the corresponding embodiments above, and the details will not be described again here.

[0117] The embodiment of the present application further provides a chip. The chip incorporates a control circuit configured to implement the functions of the image processing device 90 and one or more ports. Optionally, for functions supported by the chip, please refer to the above description. Details will not be described again here. Those skilled in the art can understand that all or part of the steps of the above embodiment can be implemented by a program instructing related hardware. The above program may be stored in a computer-readable storage medium. The above storage medium may be a read-only memory, a random access memory, etc. The processing unit or processor may be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field programmable gate array (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0118] An embodiment of the present application further provides a computer program product including instructions. When the instructions are executed on a computer, the computer can perform any of the methods in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the procedures or functions of the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or may be transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless (e.g., infrared, radio wave, or microwave) manner. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device, such as a server or data center, that includes one or more available media. The usable medium may be a magnetic medium (eg, a floppy disk, a hard disk, or a magnetic tape), an optical medium (eg, a DVD), a semiconductor medium (eg, an SSD), or the like.

[0119] It should be noted that all the above components provided in the embodiments of the present application, such as, but not limited to, memory, computer-readable storage media, and communication chips, configured to store computer instructions or computer programs are non-transitory.

[0120] All or part of the above embodiments may be implemented by using software, hardware, firmware, or any combination thereof. When a software program is used to implement the embodiments, the embodiments may be fully or partially implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed in a computer, all or part of the procedures or functions of the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or may be transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless (e.g., infrared, radio wave, or microwave) manner. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device, such as a server or data center, that includes one or more available media. The usable medium may be a magnetic medium (eg, a floppy disk, a hard disk, or a magnetic tape), an optical medium (eg, a DVD), a semiconductor medium (eg, a solid-state drive (SSD)), or the like.

[0121] Although the present application has been described with reference to embodiments, in the course of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by studying the accompanying drawings, the disclosed content, and the appended claims. In the claims, "comprising" does not exclude other components or steps, and "one" or "an" does not exclude a plurality. A single processor or other unit may implement several functions recited in the claims. Although some measures are recited in mutually different dependent claims, this does not mean that these measures cannot be combined to produce better effects.

[0122] Although the present application has been described with reference to specific features and embodiments thereof, it is clear that various modifications and combinations can be made without departing from the spirit and scope of the present application. Accordingly, the specification and the accompanying drawings are merely exemplary descriptions of the present application as defined by the appended claims, and any and all modifications, variations, combinations, or equivalents covering the scope of the present application are to be regarded as such. Obviously, a person skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. The present application is intended to cover those modifications and variations of the present application, provided that they fall within the scope of protection defined by the following claims and their equivalent technologies.

[0123] This application claims priority to Chinese Patent Application No. 202210139291.7, filed with the State Intellectual Property Office of the People's Republic of China on February 15, 2022, for the invention titled "IMAGE DYNAMIC RANGE MAPPING METHOD AND APPARATUS," and Chinese Patent Application No. 202210482398.1, filed with the State Intellectual Property Office of the People's Republic of China on May 5, 2022, for the invention titled "IMAGE PROCESSING METHOD AND APPARATUS, AND STORAGE MEDIUM," both of which are incorporated by reference in their entireties into this application.

Claims

1. 1. An image processing method, comprising: parsing an original bitstream of a target image to obtain the target image; parsing the first layered bitstream of the target image to obtain a first layered image of the target image; determining a second layered image of the image to be processed based on the image to be processed and the first layered image, the first layered image being a base layer image and the second layered image being a detail layer image, or the first layered image being a detail layer image and the second layered image being a base layer image; performing tone mapping on the base layer image to obtain a mapping result of the base layer image; fusing the mapping result of the base layer image with the detail layer image; The method according to claim 1,

2. the first layered bitstream is a downsampled bitstream; The parsing of the first layered bitstream of the target image to obtain a first layered image of the target image includes: parsing the first layered bitstream of the target image to obtain a first initial layered image of the target image; performing upsampling on the first initial layered image to obtain the first layered image; having The method of claim 1.

3. the first layered image and the second layered image are single channel images, or the first layered image and the second layered image are multi-channel images. The method according to claim 1 or 2.

4. The fusing of the mapping result of the base layer image and the detail layer image includes: obtaining a processed detail layer image based on the detail layer image and a weighting factor, the weighting factor representing a blending strength of the detail layer image; fusing the mapping result of the base layer image with the processed detail layer image; having 4. The method according to any one of claims 1 to 3.

5. Prior to parsing the original bitstream of the target image and parsing the first layered bitstream of the target image, the method further comprises: receiving the dual bit stream; the dual bitstream includes the original bitstream and the first layered bitstream; 5. The method according to any one of claims 1 to 4.

6. An image processing device, a parsing unit configured to parse an original bitstream of a target image to obtain a target image, the parsing unit further configured to parse a first layered bitstream of the target image to obtain a first layered image of the target image; a determining unit configured to determine a second layered image of the image to be processed based on the image to be processed and the first layered image, the determining unit being configured such that the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is a detail layer image and the second layered image is a base layer image; a tone mapping unit configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image; a fusion unit configured to fuse the mapping result of the base layer image and the detail layer image; An apparatus having the above configuration.

7. the first layered bitstream is a downsampled bitstream; The parsing unit includes: Parsing the first layered bitstream of the target image to obtain a first initial layered image of the target image; performing upsampling on the first initial layered image to obtain the first layered image; Specifically configured to 7. The apparatus of claim 6.

8. The first layered image and the second layered image are single channel images, or the first layered image and the second layered image are multi-channel images.

8. Apparatus according to claim 6 or 7.

9. The fusion unit is obtaining a processed detail layer image based on the detail layer image and a weighting factor representing a blending strength of the detail layer image; Fusing the mapping result of the base layer image with the processed detail layer image. Specifically configured to 7. The apparatus of claim 6.

10. The apparatus further comprises a receiving unit, the receiving unit is configured to receive a dual bit stream; the dual bitstream includes the original bitstream and the first layered bitstream; 8. Apparatus according to claim 6 or 7.

11. 1. An image processing method, comprising: receiving an original bitstream of a target image and a first layered bitstream of the target image; discarding the first layered bitstream; parsing the original bitstream to obtain the target image; performing image layering on the target image to obtain a base layer image and a detail layer image; performing tone mapping on the base layer image to obtain a mapping result of the base layer image; fusing the mapping result of the base layer image with the detail layer image; The method according to claim 1,

12. An image processing device, a receiving unit configured to receive an original bitstream of a to-be-processed image and a first layered bitstream of said to-be-processed image; a processing unit configured to discard the first layered bitstream; a parsing unit configured to parse the original bitstream to obtain the target image, the processing unit being further configured to perform image layering on the target image to obtain a base layer image and a detail layer image; a tone mapping unit configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image; a fusion unit configured to fuse the mapping result of the base layer image and the detail layer image; An apparatus having the above configuration.

13. An image processing device, A processor and an interface circuit, The processor receives or transmits data by using the interface circuit; The processor is configured to invoke program instructions stored in a memory such that the image processing device performs a method according to any one of claims 1 to 5 or claim 11. Image processing device.

14. A memory and a processor are included. the memory is configured to store program code; The processor is configured to call the program code to perform the method of any one of claims 1 to 5 or claim 11. Image processing device.

15. having a program code, When the program code is executed on a computer or a processor, the computer or the processor is capable of carrying out the method according to any one of claims 1 to 5 or claim 11. A computer-readable storage medium.

16. When executed on a computer or a processor, said computer or said processor is capable of carrying out the method according to any one of claims 1 to 5 or claim 11. Computer program products.

Images