Image processing method and device, and storage medium

Abstract

This application discloses an image processing method and apparatus, as well as a storage medium, in order to support various image layering methods, provide a flexible image fusion manner, and help improve image processing effect, and relates to the field of video image processing. The method includes: parsing an original bitstream of a target image to obtain a target image; parsing a first layered bitstream of the target image to obtain a first layered image of the target image; determining a second layered image of the target image according to the target image and the first layered image, where the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is the detail layer image and the second layered image is a base layer image; performing tone mapping on the base layer image to obtain a mapping result of the base layer image; and fusing the mapping result of the base layer image and the detail layer image.

Description

[Technical Field]

[0001] This application relates to the field of image processing, and in particular to image processing methods and apparatus, and storage media. [Background technology]

[0002] Currently, image processing techniques are widely applied in various scenarios to improve image quality. Usually, the conversion between high dynamic range (HDR) images and low dynamic range (LDR) images is performed by using image layering methods and tone mapping methods.

[0003] An image mainly contains luminance information and edge (contour) information. Luminance information determines the entire view of the image. Edge (contour) information reflects the essential attributes of objects in the image and does not change significantly with luminance. Therefore, based on the two types of information contained in the image, the image can be decomposed into a base layer image and a detail layer image, and this technique is called image layering technology.

[0004] In the field of imaging, dynamic range (DR) describes the ratio of the maximum grayscale value to the minimum grayscale value within the displayable range of an image. Natural dynamic ranges are large; for example, the luminance of a starry night scene is approximately 0.01 candela per square meter (cd / m 2 As another example, the luminance of the sun is 1,000,000,000 cd / m 2 The natural dynamic range is 10 13However, the dynamic range of the image captured by the camera is small. To ensure the imaging effect, the dynamic range needs to be reduced, and the characteristics of the HDR image, such as contrast and detail, need to be preserved as much as possible. This process is called tone mapping (TM).

[0005] In the prior art, there are no diversified methods for converting HDR images to low dynamic range (LDR). Typically, preprocessing is performed on HDR images, including converting the color space of the HDR image to CIELab and normalizing the HDR image. Image layering is performed on the preprocessed HDR images using a fast bilateral filtering method, and tone mapping and saturation adjustment are performed on the base layer image and processing is performed on the detail layer image. Finally, the processed base layer image and the processed detail layer image are combined to form a compressed LDR image. In this method, the above fixed image processing method is used in the same device, so different HDR images cannot be flexibly adjusted to achieve optimal image processing effects. Summary of the Invention

[0006] The embodiments of the present application provide an image processing method and apparatus, and a storage medium, to support various image layering methods, provide a flexible image fusion scheme, and help improve image processing effects.

[0007] To achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0008] According to a first aspect, there is provided an image processing method, the method including: parsing an original bitstream of an image to be processed to obtain a processing image; parsing a first layered bitstream of the image to be processed to obtain a first layered image of the image to be processed; determining a second layered image of the image to be processed based on the image to be processed and the first layered image, wherein the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is the detail layer image and the second layered image is the base layer image; performing tone mapping on the base layer image to obtain a mapping result of the base layer image; and fusing the mapping result of the base layer image with the detail layer image.

[0009] Current HDR image-based processing methods use fixed image layering methods, tone mapping methods, image fusion methods, etc., which prevent optimal image processing results. In this case, according to the image processing method provided herein, an image processing device parses an original bitstream and a bitstream of a first layered image to obtain a target image, obtains a second layered image based on the target image and the first layered image, and performs image processing separately on the first layered image and the second layered image. The processing methods for different layered images can be flexibly adjusted, and image fusion is performed on the flexibly processed first layered image and the second layered image, which helps improve image processing results. In other words, in this application, the image layering task during the HDR image compression process can be placed in a device other than the image processing device. However, in the prior art, image layering and the process of processing and fusing the resulting layered image are performed in the same device. The process of performing multiple image processing steps inevitably imposes limitations on the same device. The method provided herein helps achieve the effect of not restricting the specific method of image layering. Other devices send the bitstream obtained after image layering to an image processing device, which performs flexible processing and fusion based on the layered image to help improve the image processing effect of compressing HDR images.

[0010] In a possible implementation, the first layered bitstream is a downsampled bitstream, and parsing the first layered bitstream of the image to be processed to obtain a first layered image of the image to be processed comprises parsing the first layered bitstream of the image to be processed to obtain a first initial layered image of the image to be processed, and performing upsampling on the first initial layered image to obtain the first layered image.

[0011] In this possible implementation, the image processing device has an upsampling function, and the transmitted bitstream is the bitstream obtained after downsampling, i.e., the data amount of the transmitted bitstream is small, which helps to reduce the resources occupied in the bitstream transmission process, thereby reducing the bandwidth usage.

[0012] In a possible implementation, the first layered image and the second layered image are single-channel images, or alternatively, the first layered image and the second layered image are multi-channel images.

[0013] In this possible implementation, the specific format of the image is not limited, which helps to increase the flexibility and feasibility of the solution.

[0014] In a possible implementation, fusing the mapping result of the base layer image with the detail layer image includes obtaining a processed detail layer image based on the detail layer image and a weighting factor, where the weighting factor represents the overlay intensity of the detail layer image, and fusing the mapping result of the base layer image with the processed detail layer image.

[0015] In this possible implementation, a specific implementation of image fusion is provided: the processing degree of the detail layer image is flexibly changed, thereby providing personalized processing for different images and improving the image processing effect.

[0016] In a possible implementation, prior to parsing the original bitstream of the image to be processed and parsing the first layered bitstream of the image to be processed, the method further includes receiving a dual bitstream, the dual bitstream including the original bitstream and the first layered bitstream.

[0017] In this possible implementation, the image processing device performs a synchronization process based on the image to be processed and the layered image, and performs fusion on the processed image.

[0018] According to a second aspect, there is provided an image processing device including functional units configured to perform any of the methods provided in the first aspect, wherein operations performed by each function are implemented by hardware or by the hardware executing corresponding software. For example, the image processing device may include a parsing unit, a determining unit, a tone mapping unit, and a fusing unit. The parsing unit is configured to parse an original bitstream of a to-be-processed image to obtain a to-be-processed image, the parsing unit is further configured to parse a first layered bitstream of the to-be-processed image to obtain a first layered image of the to-be-processed image, the determining unit is configured to determine a second layered image of the to-be-processed image based on the to-be-processed image and the first layered image, wherein the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is the detail layer image and the second layered image is a base layer image, the tone mapping unit is configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image, and the fusing unit is configured to fuse the mapping result of the base layer image with the detail layer image.

[0019] For technical effects achieved by implementing the second aspect, please refer to the technical effects achieved by the corresponding implementation of the first aspect.

[0020] According to a third aspect, there is provided an image processing method, the method comprising: receiving an original bitstream of an image to be processed and a first layered bitstream of the image to be processed; discarding the first layered bitstream; parsing the original bitstream to obtain the image to be processed; performing image layering on the image to be processed to obtain a base layer image and a detail layer image; performing tone mapping on the base layer image to obtain a mapping result of the base layer image; and fusing the mapping result of the base layer image with the detail layer image.

[0021] Since it is believed that existing partial image processing devices do not have dual bitstream processing capabilities, by using the image processing method provided in this application, after the original bitstream and the first layered bitstream are received, the first layered bitstream can be discarded, the original bitstream is parsed to obtain a target image, and image layering, tone mapping, and fusion are performed on the target image according to existing image processing methods to obtain a fused image. The above method improves the backward compatibility and robustness of the image processing device.

[0022] According to a fourth aspect, there is provided an image processing device including functional units configured to perform any of the methods provided in the third aspect, wherein the operations performed by each function are implemented by hardware or by the hardware executing corresponding software. For example, the image processing device may include a receiving unit, a processing unit, a parsing unit, a tone mapping unit, and a fusion unit. The receiving unit is configured to receive an original bitstream of a to-be-processed image and a first layered bitstream of the to-be-processed image, the processing unit is configured to discard the first layered bitstream, the parsing unit is configured to parse the original bitstream to obtain the to-be-processed image, the processing unit is further configured to perform image layering on the to-be-processed image to obtain a base layer image and a detail layer image, the tone mapping unit is configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image, and the fusion unit is configured to fuse the mapping result of the base layer image and the detail layer image.

[0023] For technical effects achieved by implementing the fourth aspect, please refer to the technical effects achieved by the corresponding implementation of the third aspect.

[0024] According to a fifth aspect, there is provided an image processing apparatus, the apparatus comprising a processor and an interface circuit, the processor receiving or transmitting data by using the interface circuit, the processor being configured to invoke program instructions stored in a memory such that the image processing apparatus performs any of the methods provided in the first or third aspects.

[0025] According to a sixth aspect, there is provided an image processing apparatus, the apparatus including a processor and a memory, the processor coupled to the memory, the memory configured to store computer-executable instructions, the processor executing the computer-executable instructions stored in the memory to perform any of the methods provided in the first or third aspect.

[0026] According to a seventh aspect, there is provided a chip, the chip including a processor and an interface circuit, the interface circuit configured to receive code instructions and communicate the code instructions to the processor, the processor configured to execute the code instructions to perform any of the methods provided in the first or third aspects.

[0027] According to an eighth aspect, there is provided a computer-readable storage medium, comprising computer-executable instructions which, when executed on a computer, enable the computer to perform any of the methods provided in the first or third aspect.

[0028] According to a ninth aspect, there is provided a computer program product, comprising computer-executable instructions which, when executed on a computer, enable the computer to perform any of the methods provided in the first or third aspect.

[0029] For the technical effects achieved by any of the implementations of the fifth to ninth aspects, please refer to the technical effects achieved by the corresponding implementations of the first or third aspects, and the details will not be described again here. [Brief explanation of the drawings]

[0030] [Figure 1] 1 is a diagram of the structure of an image processing device according to an embodiment of the present application; [Figure 2] 1 is a diagram of the structure of a video coding device according to an embodiment of the present application; [Figure 3]1 is a schematic flow chart of an image capture device and an image processing apparatus according to an embodiment of the present application. [Figure 4] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Figure 5] FIG. 1 is a diagram of a local tone mapping method according to an embodiment of the present application; [Figure 6a] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Figure 6b] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Figure 7] 1 is a schematic flow chart of a multi-channel image processing method according to an embodiment of the present application; [Figure 8] 1 is a schematic flowchart of a downsampling image processing method according to an embodiment of the present application; [Figure 9] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Figure 10] 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. [Figure 11] 1 is a diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present application. [Figure 12] 1 is a diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0031] In the present specification, unless otherwise specified, " / " means "or." For example, A / B may represent A or B. The term "and / or" in this specification simply indicates an association relationship between related objects, and indicates that three relationships may exist. For example, A and / or B may represent the following three cases: only A exists, both A and B exist, or only B exists. Furthermore, "at least one" means one or more, and "multiple" means two or more. Terms such as "first" and "second" do not limit the quantity and execution order, and terms such as "first" and "second" do not indicate clear distinction.

[0032] It should be noted that terms such as "example" or "for example" are used herein to denote serving as an example, illustration, or description. Any embodiment or design scheme described herein as an "example" or "for example" should not be described as preferred or having more advantages than other embodiments or design schemes. Indeed, the use of terms such as "example," "for example," and the like is intended to present related concepts in a concrete manner.

[0033] An embodiment of the present application provides an image processing method that can be applied to an image processing device. The specific form of the image processing device is not limited in this embodiment of the present application. For example, the image processing device may specifically be a terminal device. The terminal device may be referred to as a terminal, user equipment (UE), terminal device, video coding device, access terminal, subscriber unit, subscriber station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication device, user agent, user equipment, etc. The terminal device may specifically be a mobile phone, a camera, a display, an augmented reality (AR) device, a virtual reality (VR) device, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc.

[0034] In the present application, the structure of an image processing device may be shown in Fig. 1. Fig. 1 is a diagram of the hardware structure of an image processing device 10 according to an embodiment of the present application. The image processing device 10 may be configured to implement the functions of the image processing device.

[0035] 1 may include a processor 101, a memory 102, a communication interface 103, and a bus 104. The processor 101, the memory 102, and the communication interface 103 may be connected via the bus 104.

[0036] The processor 101 is the control center of the image processing device 10 and may be a general-purpose central processing unit (CPU), other general-purpose processor, etc. The general-purpose processor may be a microprocessor, any conventional processor, etc.

[0037] For example, processor 101 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG.

[0038] Memory 102 may be, but is not limited to, read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, electrically erasable programmable read-only memory (EEPROM), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying expected program code in the form of instructions or data structures and accessible by a computer.

[0039] In a possible implementation, the memory 102 may be independent of the processor 101. The memory 102 may be connected to the processor 101 via a bus 104 and is configured to store data, instructions, or program codes. When calling and executing the instructions or program codes stored in the memory 102, the processor 101 can implement the image processing methods provided in the embodiments of the present application.

[0040] In another possible implementation, the memory 102 may alternatively be integrated with the processor 101 .

[0041] The communication interface 103 is configured to connect the image processing device 10 to other devices by using a communication network. The communication network may be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 103 may include a receiving unit configured to receive data and a transmitting unit configured to transmit data.

[0042] The bus 104 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. Buses may be classified as address buses, data buses, control buses, etc. For simplicity of representation, only one bold line is used in FIG. 1 for representation, but this does not represent the presence of only one bus or one type of bus.

[0043] It should be noted that the structure shown in Figure 1 does not constitute a limitation on the image processing device 10. In addition to the components shown in Figure 1, the image processing device 10 may include more or fewer components than those shown, may combine some components, or may have a different component arrangement.

[0044] For example, when the image processing device is a video coding device, the processor further includes a coding module. FIG. 2 is a diagram of a hardware structure of a video coding device 200 according to an embodiment of the present application. The video coding device may be a decoding device or an encoding device. The video coding device 200 is suitable for implementing the disclosed examples / embodiments as described herein. The video coding device 200 includes a downlink port 220, an uplink port 250, and / or a transceiver unit (Tx / Rx) 210. The transceiver unit 210 includes a transmitter and / or a receiver configured to perform uplink and / or downlink data communication over a network. The video coding device 200 further includes a processor 230 and a memory 240. Video coding device 200 may further include electrical, optical-to-electrical (OE), electrical-to-optical (EO), and / or wireless communication components coupled to uplink port 250 and / or downlink port 220 for communicating data over an electrical, optical, or wireless communication network. Video coding device 200 may further include input and / or output (I / O) devices 260 for communicating data with a user. I / O devices 260 may include output devices such as a display for displaying video data and speakers for outputting audio data. I / O devices 260 may also include input devices such as a keyboard, mouse, trackball, and / or corresponding interfaces for interacting with such output devices.

[0045] The processor 230 is implemented in hardware and software. The processor 230 may be implemented as one or more CPU chips, one or more cores (e.g., implemented as a multi-core processor), one or more field-programmable gate arrays (FPGAs), one or more application-specific integrated circuits (ASICs), and one or more digital signal processors (DSPs). The processor 230 communicates with the downlink port 220, the Tx / Rx 210, the uplink port 250, and the memory 240. The processor 230 includes a coding module 232. The coding module 232 performs the bitstream parsing steps in the following disclosed embodiments, such as steps S201 and S202 shown in FIG. 4. The coding module 232 may also perform any other method / mechanism described herein. Additionally, the processor 230 may further include other modules configured to process the image parsed by the coding module 232, with multiple modules working together to complete the image processing methods provided herein.

[0046] It should be noted that a video coding device is configured to implement video encoding / decoding functionality, which may alternatively be implemented by using other types of devices, such as decoding circuits.

[0047] Currently, common methods for image layering include image layering using filters or image layering using artificial intelligence (AI) methods. Image layering is performed by using filters, and the base layer image after image layering is obtained by extracting low-frequency information from the image, where the low-frequency information is an area with low changes in grayscale values, and the detail layer image after image layering is obtained by extracting high-frequency information from the image, where the high-frequency information is an area with high changes in grayscale values. It can be understood that areas with low changes in grayscale values ​​represent the gradient state of the image, and areas with high changes in grayscale values ​​represent clear differences in the image, such as the contours of objects. When filters are used to perform image layering, hardware deployment is required in the image processing device. This method is costly and complex to implement, and maintenance and upgrades are inconvenient. When AI methods are used to perform image layering, high computing power is required in the image processing device, and networks with excellent layering effects but large scales cannot be deployed using this method. In conclusion, in this application, the image layering task is placed on another device, and the specific method for image layering is not limited, so that the image processing device can flexibly set the image processing method, and different HDR images can be flexibly adjusted during image processing to achieve optimal effects. As shown in FIG. 3, the capture device acquires an HDR image, performs pre-production on the HDR image, such as image pre-processing, and then performs image layering. The image layering method is not limited in this application. A portion of the layered image and the original image are compressed into a bitstream and transmitted to the image processing device provided in this application. The image processing device parses the bitstream and performs image processing on the layered image separately, and then obtains an LDR image through image fusion to compress the HDR image. Flexible image processing and fusion methods are used to improve the compression effect of HDR images. The technical solutions provided in the embodiments of this application are described below with reference to the accompanying drawings.

[0048] 4 is a flowchart of the image processing method according to the present application. The method includes the following steps:

[0049] S201: The image processing device parses the original bitstream of the target image to obtain the target image.

[0050] The original bitstream is a data stream that is sent to the image processing device after the target image has been compressed, and the image processing device obtains the target image by parsing the original bitstream.

[0051] S202: The image processing device parses the first layered bitstream of the image to be processed to obtain a first layered image of the image to be processed.

[0052] The first layered bitstream is a data stream in which an image at an arbitrary layer after image layering has been performed on the image to be processed is compressed and transmitted to the image processing device. The image processing device obtains the first layered image by parsing the first layered bitstream.

[0053] The first layered image is a base layer image or a detail layer image.

[0054] Optionally, the first layered image is a single-channel image or a multi-channel image. The number of image channels is determined based on the image color mode. For example, when the image color mode is a red, green, blue (RGB) mode, it indicates that the image is formed by parameters of three channels: R, G, and B, and the image may be called a three-channel image. As another example, when the image mode is a grayscale mode, the image is formed by parameters of one channel, and the image may be called a single-channel image. Furthermore, the multi-channel image may also include a dual-channel image, a four-channel image, etc.

[0055] It should be noted that the image to be processed may be a single-channel image or a multi-channel image. When the image to be processed is a single-channel image, the first layered image is a single-channel image. When the image to be processed is a multi-channel image, the first layered image is a single-channel image or a multi-channel image. Specifically, it is assumed that the number of channels of the image to be processed is N, the number of channels of the first layered image is M, N is an integer greater than 1, and 1≦M≦N.

[0056] Optionally, the first layered bitstream is a downsampled bitstream. Step S202 specifically includes: the image processing device parsing the first layered bitstream of the image to be processed to obtain a first initial layered image of the image to be processed. The image processing device performs upsampling on the first initial layered image to obtain the first layered image.

[0057] The first initial layered image is a downsampled image, and the image on which upsampling is performed is an upsampled image. The downsampled image is an image output after zooming out, and the upsampled image is an image output after zooming in.

[0058] If the first layered bitstream is a downsampled bitstream, the image processing device parses the first layered bitstream to obtain a first initial layered image, and performs upsampling on the first initial layered image, in other words, zooms in on the first initial layered image to obtain the first layered image, which has the same resolution as that of the image to be processed, thereby obtaining a second layered image thereafter.

[0059] Optionally, before step S201 and step S202 are performed, the method further includes step S200: the image processing device receives the original bitstream and the first layered bitstream.

[0060] The original bitstream and the first layered bitstream are transmitted to the image processing device as a single bitstream or a dual bitstream. The encapsulation format of the single bitstream is different from that of the dual bitstream. When the original bitstream and the first layered bitstream are transmitted to the image processing device in the form of a dual bitstream, the image processing device receives the dual bitstream, which includes the original bitstream of the image to be processed and the first layered bitstream of the image to be processed. The image processing device may synchronously parse the original bitstream and the first layered bitstream to obtain the image to be processed and the first layered image and perform an image processing process. When the original bitstream and the first layered bitstream are transmitted to the image processing device in the form of a single bitstream, the bitstream may further include synchronization information, which is used by the image processing device to determine corresponding positions of pixels in the image to be processed and the first layered image to perform image processing processes such as obtaining another layered image or performing image fusion.

[0061] It can be understood that when the original bitstream and the first layered bitstream are transmitted to the image processing device in the form of a dual bitstream, the image processing device performs synchronization processing based on the image to be processed and the layered image, and performs fusion on the processed image, thereby shortening the image processing time.

[0062] It should be noted that the present application does not strictly require a method for parsing the original bitstream and the first layered bitstream. When the original bitstream and the first layered bitstream are transmitted to the image processing device in the form of a dual bitstream, the image processing device may receive the dual bitstream and parse the dual bitstream synchronously, or may parse the original bitstream and the first layered bitstream separately according to sequence.

[0063] S203: The image processing device determines a second layered image of the image to be processed based on the image to be processed and the first layered image, where the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is a detail layer image and the second layered image is a base layer image.

[0064] It can be understood that image layering is performed on the to-be-processed image to obtain a base layer image and a detail layer image. Thus, the to-be-processed image includes the base layer image and the detail layer image. The image processing device obtains the second layered image by calculating the difference between the to-be-processed image and the first layered image.

[0065] Optionally, the image processing device calculates pixel differences between pixels of the image to be processed and the first layered image, and uses the image formed by the pixel differences as the second layered image.

[0066] Optionally, the image processing device calculates the difference between the original bitstream and the first layered bitstream, and parses the calculated difference bitstream to obtain the second layered image.

[0067] It can be understood that the first layered image and the second layered image have the same number of channels. If the first layered image is a single-channel image, the second layered image is a single-channel image. If the first layered image is a multi-channel image, the second layered image is a multi-channel image with the same number of channels.

[0068] Currently, for HDR images, tone mapping is mainly performed on the luminance information of an image, and then the edge information of the image is fused to obtain a fused LDR image. The contrast of an image reflects the degree of brightness in the image, and the degree of brightness is determined based on the luminance information. To restore the contrast in an HDR image, tone mapping is generally performed on the luminance information of the image. To restore detail features in an HDR image, image processing is generally performed on the edge information of the image. A similar effect to that in an HDR image can be achieved by fusing the luminance information and edge information, so that images with similar effects can be presented to users on different hardware devices.

[0069] Specifically, the image processing device calculates luminance information Y source and extracting luminance information Y1 in the first layered image using the following formula: Y2=Y source -Y1 The luminance information Y2 of the second layered image is obtained by calculation according to the following formula.

[0070] Here, the first layered image is a base layer image, and the second layered image is a detail layer image. In this case, Y1 is Y base and Y2 may be expressed as Y detail Alternatively, the first layered image is a detail layer image, and the second layered image is a base layer image, and Y1 is Y detail and Y2 can be expressed as Y base It may be expressed as:

[0071] It can be understood that the luminance information is in units of pixels, and the luminance information of each pixel is represented by using a pixel value. In the calculation process, the pixel value of each pixel is used for calculation.

[0072] S204: The image processing device performs tone mapping on the base layer image to obtain a mapping result of the base layer image.

[0073] The specific method of tone mapping is not limited in the present application. Tone mapping includes global tone mapping (GTM), local tone mapping, etc. ( There are local tone mapping (LTM) and AI tone mapping based on deep neural networks. GTM means that pixel values ​​of all pixels in a base layer image are mapped using the same mapping function. LTM means that pixel values ​​of pixels in different areas of the base layer image are mapped using different mapping functions. LTM may divide an image into multiple image blocks, and the same or different mapping functions are used for the multiple image blocks. For example, as shown in Figure 5, an image is divided into multiple image blocks, and the same mapping function is used for image blocks filled with the same format. For example, a shaded image block means that mapping function 2 is used for pixel values ​​in that image block, and a black image block means that mapping function 3 is used for pixel values ​​in that image block.

[0074] It should be noted that the image processing device may use different tone mapping methods for different images to be processed, and the image processing device may be selected according to the tone mapping methods supported by the device.

[0075] Optionally, if the image processing device supports at least two tone mapping methods, the image processing device performs image processing by using multiple tone mapping methods on the same processing target image, compares the processing effects of different tone mapping methods, and selects the processing result with the best effect.

[0076] S205: The image processing device fuses the mapping result of the base layer image with the detail layer image.

[0077] Specifically, the image processing device overlaps the mapping result of the base layer image with the detail layer image to obtain a fused image.

[0078] Optionally, the image processing device obtains a processed detail layer image based on the detail layer image and a weighting factor, where the weighting factor represents a blending strength of the detail layer image, and the blending strength indicates a proportion of the processed detail layer image in the fused image. The image processing device blends the mapping result of the base layer image with the processed detail layer image.

[0079] It can be understood that the current detail layer image contains high-frequency information in the image. After the detail layer image is processed based on different weighting coefficients, the high-frequency information in the image is actually enhanced or weakened, so that the mapping results of the processed detail layer image and the base layer image have different effects during image fusion. In other words, the different weighting coefficients help to enhance or weaken the high-frequency information in the image, so as to indirectly affect the proportion of high-frequency information in the fused image in the image fusion process.

[0080] Specifically, the processed detail layer image is Y' detail where the weighting coefficient is weight and Y' detail is calculated in the following way: Y' detail =Y detail ×weight

[0081] where the weight is a constant greater than 0. For example: weight=α

[0082] where α>0. The image processing device blends pixel values ​​of the mapping result of the base layer image with pixel values ​​in the detail layer image with the same overlapping strength, i.e., weight=((Y tm / Y base )+α) β

[0083] where α>0 and β>0. The image processing device separately determines a weighting factor for fusion with the detail layer image based on pixel values ​​of the mapping result of the base layer image. Specifically, the image processing device determines a ratio of each pixel value of the mapping result of the base layer image to a pixel value at a corresponding position in the base layer image, and determines a weighting factor for the pixel based on the ratio.

[0084] After the image processing device determines the processed detail layer image, the processed detail layer image is mapped to the mapping result Y tm and the fused image Y is given by: Y=Y tm +Y' detail is calculated by using

[0085] Alternatively, Y is: Y=Y tm +Y detail ×weight It can be expressed as:

[0086] Through steps S201 to S205, the image processing device performs more flexible processing on the image. The image processing device obtains the processing target image and several layered images, performs various processing on the layered images, and adaptively adjusts the image fusion effect, which is helpful to improve the image processing effect and user experience.

[0087] Steps S201 to S205 can be specifically implemented by using the flowchart shown in FIG. 6a or FIG. 6b.

[0088] If the first layered bitstream is a bitstream of a base layer image, please refer to the image processing process shown in Figure 6a. If the first layered bitstream is a bitstream of a detail layer image, please refer to the image processing process shown in Figure 6b.

[0089] Optionally, if the image obtained by the image processing device parsing the first layered bitstream is a multi-channel image (e.g., a three-channel image), please refer to the image processing process shown in Figure 7.

[0090] where: weight1=α1, weight2=α2, and weight3=α3, where α1, α2, and α3 are all constants greater than zero, or, weight1=((C tm1 / C base1 )+α1) β1 weight2=((C tm2 / C base2 )+α2) β2 , and weight3=((C tm3 / C base2 )+α3) β3 and where α1, α2, α3, β1, β2, and β3 are all constants greater than zero.

[0091] Accordingly, the image processing device: C1=C tm1 +C' detail1 , C2=C tm2 +C' detail2 , and C3=C tm3 +C' detail3 Calculate the pixel values ​​of the fused image as follows:

[0092] where: C' detail =C detail ×weight

[0093] So the pixel values ​​of the fused image are: C1=C tm1 +C detail1 ×weight1, C2=C tm2 +C detail2 ×weight2, C3=C tm3 +C detail3 ×weight3 It can also be expressed as:

[0094] It can be understood that when the image to be processed and the first layered image are both multi-channel images, image processing can be performed independently on each channel, and the fused image obtained by image processing is also a multi-channel image.

[0095] 7 uses an example in which the first layered image is a base layer image, but the first layered image may alternatively be a detail layer image. In the image processing process shown in FIG. 6b, the relevant specific parameters are the same as those in FIG. 7. The details will not be described again.

[0096] Optionally, if the image obtained by the image processing device parsing the first layered bitstream is a downsampled first layered image, the following uses the downsampled base layer image as an example. As shown in Figure 8, the image processing device performs upsampling on the downsampled base layer image to obtain a base layer image, and then performs subsequent image processing processes.

[0097] When the bitstream is a downsampled first layered bitstream and transmitted to the image processing device, the transmission bandwidth is saved and the image processing efficiency is improved.

[0098] Optionally, other image processing and fusion methods after layering are provided. As shown in Figure 9, tone mapping and image fusion are performed synchronously, and a fused image is output. This allows for more flexible selection of the overlapping position between the detail layer image obtained by tone mapping and the base layer image, which is expected to enable expansion of the device path design of the image processing device, improvement of the feasibility of achieving the optimal effect of tone mapping, and ease of implementation by ensuring compatibility with existing image processing devices.

[0099] Optionally, if it is considered that the existing partial image processing device does not have the dual bitstream processing capability, as shown in FIG. 10, after receiving the original bitstream and the first layered bitstream, the image processing device may discard the first layered bitstream and use the original bitstream as Parse and obtain a target image by performing image layering, tone mapping, and fusion on the target image according to an existing image processing method to obtain a fused image. The above method improves the backward compatibility and robustness of the image processing device device.

[0100] Optionally, the bitstream may be non-gamma (gamma) coded; common formats include perceptual quantization (PQ) and hybrid log-gamma (HLG).

[0101] The above mainly describes the solutions in the embodiments of the present application from the perspective of a method. To implement the above functions, the image processing device includes at least one corresponding hardware structure and corresponding software module for performing each function. Those skilled in the art will easily realize that the present application can be implemented by hardware or a combination of hardware and computer software, in combination with the example units and algorithm steps described in the embodiments disclosed herein. Whether the functions are implemented by hardware or hardware driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but it should not be considered that such implementation goes beyond the scope of the present application.

[0102] In the embodiment of the present application, the division into functional units may be performed in the image processing device based on the above example of the method. For example, the functional units may be divided based on corresponding functions, or two or more functions may be integrated into one processing unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit. Note that in this embodiment of the present application, the division into units is merely an example and is merely a logical division of functions. In actual implementation, other division methods may be used.

[0103] For example, Figure 11 is a possible diagram of the structure of the image processing device (denoted as image processing device 90) in the above embodiment. The image processing device 90 includes a parsing unit 901, a determining unit 902, a tone mapping unit 903, and a fusing unit 904. The parsing unit 901 is configured to parse the original bitstream of the target image to obtain the target image, which teeth 4, for example, step S201 shown in FIG. 4. The parsing unit 901 is further configured to parse the first layered bitstream of the image to be processed to obtain a first layered image of the image to be processed, for example, step S202 shown in FIG. 4. The determining unit 902 is configured to determine a second layered image of the image to be processed based on the image to be processed and the first layered image, where the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is the detail layer image and the second layered image is the base layer image, for example, step S203 shown in FIG. 4. The tone mapping unit 903 is configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image, for example, step S204 shown in FIG. 4. The fusion unit 904 is configured to fuse the mapping result of the base layer image and the detail layer image, for example, step S205 shown in FIG. 4.

[0104] Optionally, the first layered bitstream is a downsampled bitstream, and the parsing unit 901 is particularly configured to parse the first layered bitstream of the image to be processed to obtain a first initial layered image of the image to be processed, and to perform upsampling on the first initial layered image to obtain the first layered image.

[0105] Optionally, the first layered image and the second layered image are single-channel images, or the first layered image and the second layered image are multi-channel images.

[0106] Optionally, the fusion unit 904 is specifically configured to obtain a processed detail layer image based on the detail layer image and a weighting factor representing the overlapping strength of the detail layer image, and fuse the mapping result of the base layer image with the processed detail layer image.

[0107] Optionally, the image processing device 90 further includes a receiving unit 905, where the receiving unit 905 is configured to receive a dual bitstream, where the dual bitstream includes an original bitstream and a first layered bitstream.

[0108] Optionally, the image processing device 90 further includes a storage unit 906. The storage unit 906 is configured to store computer-executable instructions, and other units in the image processing device can perform corresponding operations based on the computer-executable instructions stored in the storage unit 906.

[0109] For specific descriptions of any of the above aspects, please refer to the above method embodiments, and the details will not be repeated here. Furthermore, for any descriptions of the image processing device 90 provided above and descriptions of the advantageous effects, please refer to the above corresponding method embodiments, and the details will not be repeated here.

[0110] In an example, referring to Figure 1, some or all of the functions performed by the parsing unit 901, the determining unit 902, the tone mapping unit 903, the fusing unit 904, the receiving unit 905, and the storage unit 906 in the image processing device 90 may be implemented by the processor 101 in Figure 1 by executing program code in the memory 102 in Figure 1. Alternatively, the receiving unit 905 may be implemented by using the receiving unit in the communication interface 103 in Figure 1.

[0111] For example, Fig. 12 is a possible diagram of the structure of an image processing device (denoted as image processing device 100) in the above embodiment. The image processing device 100 includes a receiving unit 1001 configured to receive an original bitstream of a processing target image and a first layered bitstream of the processing target image, a processing unit 1002 configured to discard the first layered bitstream, and a processing unit 1003 configured to convert the original bitstream to obtain the processing target image. Parse the processing unit 1002 includes a parsing unit 1003 configured to perform image layering on the image to be processed to obtain a base layer image and a detail layer image, the parsing unit 1003 being further configured to perform image layering on the image to be processed to obtain a base layer image and a detail layer image, a tone mapping unit 1004 configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image, and a fusion unit 1005 configured to fuse the mapping result of the base layer image and the detail layer image.

[0112] Optionally, the image processing device 100 further includes a storage unit 1006. The storage unit 1006 is configured to store computer-executable instructions, and other units in the image processing device may perform corresponding operations based on the computer-executable instructions stored in the storage unit 1006.

[0113] For specific descriptions of any of the above aspects, please refer to the above method embodiments, and the details will not be described again here. Furthermore, for any descriptions of the image processing device 100 provided above and descriptions of the advantageous effects, please refer to the above corresponding method embodiments, and the details will not be described again here.

[0114] In an example, referring to Figure 1, some or all of the functions performed by the receiving unit 1001, the processing unit 1002, the parsing unit 1003, the tone mapping unit 1004, the fusing unit 1005, and the storage unit 1006 in the image processing device 100 may be implemented by the processor 101 in Figure 1 by executing program code in the memory 102 in Figure 1. Alternatively, the receiving unit 1001 may be implemented by using a receiving unit in the communication interface 103 in Figure 1.

[0115] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, which, when executed by a computer, causes the computer to perform the method performed by the image processing device provided above.

[0116] For the description of the related contents and advantageous effects of any computer-readable storage medium provided above, please refer to the corresponding embodiments above, and the details will not be described again here.

[0117] An embodiment of the present application further provides a chip. The chip incorporates a control circuit configured to implement the functions of the image processing device 90 and one or more ports. Optionally, for functions supported by the chip, please refer to the above description. Details will not be described again here. Those skilled in the art will understand that all or part of the steps of the above embodiment can be implemented by a program instructing related hardware. The program may be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a random-access memory, etc. The processing unit or processor may be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field programmable gate array (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0118] An embodiment of the present application further provides a computer program product including instructions. When the instructions are executed on a computer, the computer can perform any of the methods in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the procedures or functions of the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless (e.g., infrared, radio wave, or microwave) method. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device, such as a server or data center, that includes one or more available media. The usable media may be magnetic media (eg, floppy disk, hard disk, or magnetic tape), optical media (eg, DVD), semiconductor media (eg, SSD), and the like.

[0119] It should be noted that all the above components provided in the embodiments of the present application, such as, but not limited to, memory, computer-readable storage media, and communication chips, configured to store computer instructions or computer programs, are non-transitory.

[0120] All or part of the above embodiments may be implemented using software, hardware, firmware, or any combination thereof. When a software program is used to implement the embodiments, the embodiments may be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the procedures or functions of the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless (e.g., infrared, radio wave, or microwave) transmission. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device, such as a server or data center, that includes one or more available media. The available media may be magnetic media (for example, a floppy disk, a hard disk, or a magnetic tape), optical media (for example, a DVD), semiconductor media (for example, a solid-state drive (SSD)), or the like.

[0121] Although the present application has been described with reference to embodiments, in the course of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments, by studying the accompanying drawings, the disclosed content, and the appended claims. In the claims, "comprising" does not exclude other components or steps, and "one" or "an" does not exclude a plurality. A single processor or other unit may implement several functions recited in the claims. Although certain measures are recited in mutually different dependent claims, this does not indicate that these measures cannot be combined to produce better effects.

[0122] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made without departing from the spirit and scope of the present application. Accordingly, the specification and accompanying drawings are merely exemplary descriptions of the present application as defined by the appended claims, and any and all modifications, variations, combinations, or equivalents covering the scope of the present application are to be considered. Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. The present application is intended to cover those modifications and variations of the present application, provided that they fall within the scope of protection defined by the following claims and their equivalent technologies.

[0123] This application claims priority to Chinese Patent Application No. 202210139291.7, filed with the State Intellectual Property Office of the People's Republic of China on February 15, 2022, entitled "IMAGE DYNAMIC RANGE MAPPING METHOD AND APPARATUS," and Chinese Patent Application No. 202210482398.1, filed with the State Intellectual Property Office of the People's Republic of China on May 5, 2022, entitled "IMAGE PROCESSING METHOD AND APPARATUS, AND STORAGE MEDIUM," both of which are incorporated herein by reference in their entirety.

Claims

1. 1. An image processing method for converting a high dynamic range image into a low dynamic range image, comprising: parsing the original bitstream of the target image to obtain the target image; parsing the first layered bitstream of the image to be processed to obtain a first layered image of the image to be processed; determining a second layered image of the image to be processed based on the image to be processed and the first layered image, wherein the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is a detail layer image and the second layered image is a base layer image; performing tone mapping on the base layer image to obtain a mapping result of the base layer image; fusing the mapping result of the base layer image and the detail layer image to obtain the low dynamic range image; A method having the following.

2. the first layered bitstream is a downsampled bitstream; Parsing the first layered bitstream of the target image to obtain the first layered image of the target image includes: parsing the first layered bitstream of the image to be processed to obtain a first initial layered image of the image to be processed; performing upsampling on the first initial layered image to obtain the first layered image; having The method of claim 1.

3. the first layered image and the second layered image are single-channel images, or the first layered image and the second layered image are multi-channel images. The method of claim 1.

4. Fusing the mapping result of the base layer image with the detail layer image includes: determining a weighting factor based on the mapping result of the base layer image by determining a ratio of each pixel of the mapping result of the base layer image to a pixel value at a corresponding position in the base layer image, and determining a weighting factor for that pixel based on the ratio; obtaining a processed detail layer image based on the detail layer image and the weighting factor, the weighting factor representing a blending strength of the detail layer image; fusing the mapping result of the base layer image with the processed detail layer image; having The method of claim 1.

5. Prior to parsing the original bitstream of the image to be processed and the first layered bitstream of the image to be processed, the method further comprises: receiving a dual bitstream; the dual bitstream includes the original bitstream and the first layered bitstream; The method of claim 1.

6. 1. An image processing device for converting a high dynamic range image into a low dynamic range image, comprising: a parsing unit configured to parse an original bitstream of the to-be-processed image to obtain a to-be-processed image, the parsing unit further configured to parse a first layered bitstream of the to-be-processed image to obtain a first layered image of the to-be-processed image; a determination unit configured to determine a second layered image of the image to be processed based on the image to be processed and the first layered image, the determination unit being configured such that the first layered image is a base layer image and the second layered image is a detail layer image, or the first layered image is a detail layer image and the second layered image is a base layer image; a tone mapping unit configured to perform tone mapping on the base layer image to obtain a mapping result of the base layer image; a fusion unit configured to fuse the mapping result of the base layer image and the detail layer image to obtain the low dynamic range image; A device having:

7. the first layered bitstream is a downsampled bitstream; The parsing unit parsing the first layered bitstream of the target image to obtain a first initial layered image of the target image; performing upsampling on the first initial layered image to obtain the first layered image; It is configured as follows:

7. The apparatus of claim 6.

8. the first layered image and the second layered image are single-channel images, or the first layered image and the second layered image are multi-channel images; 7. The apparatus of claim 6.

9. The fusion unit is determining a weighting factor based on the mapping result of the base layer image by determining a ratio of each pixel of the mapping result of the base layer image to a pixel value at a corresponding position in the base layer image, and determining a weighting factor for the pixel based on the ratio; obtaining a processed detail layer image based on the detail layer image and the weighting factor representing a blending strength of the detail layer image; fusing the mapping result of the base layer image with the processed detail layer image. It is configured as follows:

7. The apparatus of claim 6.

10. The device further comprises a receiving unit; the receiving unit is configured to receive a dual bitstream; the dual bitstream includes the original bitstream and the first layered bitstream; 7. The apparatus of claim 6.

11. An image processing device, a processor and an interface circuit; the processor receives or transmits data by using the interface circuit; The processor is configured to invoke program instructions stored in memory to cause the image processing device to perform the method of any one of claims 1 to 5. Image processing device.

12. a memory and a processor; the memory is configured to store program code; The processor is configured to call the program code to perform the method of any one of claims 1 to 5. Image processing device.

13. When executed on a computer or a processor, said computer or said processor is capable of carrying out the method of any one of claims 1 to 5. Computer program.

Images