High dynamic range image processing method, electronic device, and storage medium

Abstract

The application provides a high dynamic range image processing method, an electronic device and a storage medium, and relates to the technical field of computer vision and digital image processing. The method comprises the following steps: obtaining original image data; inputting the original image data into a high dynamic range conversion model which is pre-trained; calculating luminance prior data based on the original image data by a luminance estimation module in the high dynamic range conversion model; generating luminance features and a luminance enhanced image based on the luminance prior data and the original image data by the luminance estimation module; processing the luminance enhanced image and the luminance features by a luminance guiding module in the high dynamic range conversion model to obtain a refined luminance image; processing the refined luminance image by a chroma refining module in the high dynamic range conversion model to output high dynamic range image data. The final high dynamic range image data can reach a cinema-level standard in terms of color accuracy, visual naturalness and artistic expressiveness.

Description

Technical Field

[0001] This application relates to the fields of computer vision and digital image processing technology, and in particular to a high dynamic range image processing method, electronic device and storage medium. Background Technology

[0002] Existing SDR (Standard Dynamic Range) to HDR (High Dynamic Range) conversion methods are mostly designed for consumer-grade content such as television, and adopt a global tone mapping strategy. They ignore the stringent requirements of cinema-grade scenes for fine brightness details (such as highlights and shadows) and color accuracy. As a result, the converted HDR content is prone to problems such as loss of detail and skin distortion in the darkroom environment of a cinema, and cannot meet the professional cinema projection standards. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a high dynamic range image processing method, electronic device and storage medium to solve the problems of detail loss and skin color distortion that easily occur in existing converted HDR images.

[0004] To achieve the above objectives, the first aspect of this application provides a high dynamic range image processing method, comprising: Obtain the raw image data; The original image data is input into a pre-trained high dynamic range conversion model. The brightness estimation module in the high dynamic range conversion model calculates the brightness prior data based on the original image data. Based on the brightness prior data and the original image data, the brightness estimation module generates brightness features and a brightness-enhanced image. The brightness enhancement image and the brightness features are processed by the brightness guidance module in the high dynamic range conversion model to obtain a refined brightness image; The refined luminance image is processed by the chroma refinement module in the high dynamic range conversion model to output high dynamic range image data.

[0005] Optionally, the brightness estimation module includes a first convolutional layer, a depth-separable convolutional layer, and a second convolutional layer; The step of generating brightness features and a brightness-enhanced image through the brightness estimation module based on the prior brightness data and the original image data includes: The original image data and the brightness prior data are concatenated by channels to obtain the concatenated features; The splicing features are input into the first convolutional layer, and the fused features are output through the first convolutional layer. The fused features are input into the depthwise separable convolutional layer, and the brightness features are obtained by feature extraction through the depthwise separable convolutional layer. The brightness features are input into the second convolutional layer, and the brightness scaling map is output through the second convolutional layer; The brightness-enhanced image is generated based on the brightness scaling map and the brightness prior data.

[0006] Optionally, the step of processing the brightness enhancement image and the brightness features through the brightness guidance module in the high dynamic range conversion model to obtain a refined brightness image includes: Based on the brightness features, the brightness enhancement image is downsampled at multiple scales by the brightness guidance module to generate a multi-scale feature map. The multi-scale feature map is upsampled and the residual is fused to generate the refined brightness image.

[0007] Optionally, the brightness guidance module includes a third convolutional layer, a first brightness guidance attention submodule, a fourth convolutional layer, a second brightness guidance attention submodule, and a fifth convolutional layer; the multi-scale feature map includes a first feature map, a second feature map, and a third feature map; The step of performing multi-scale feature downsampling on the brightness-enhanced image based on the brightness features, through the brightness guidance module, to generate a multi-scale feature map includes: The brightness-enhanced image is input into the third convolutional layer, and the first feature map is generated through the third convolutional layer; The first feature map is processed sequentially through the first brightness-guided attention submodule and the fourth convolutional layer to generate the second feature map; The second feature map is processed sequentially through the second brightness-guided attention submodule and the fifth convolutional layer to generate the third feature map; wherein the brightness features are applied to the brightness-guided multi-head self-attention submodules in the first brightness-guided attention submodule and the second brightness-guided attention submodule, respectively.

[0008] Optionally, the step of performing feature upsampling and residual fusion on the multi-scale feature map to generate the refined brightness image includes: The multi-scale feature map is upsampled and fused with residuals to generate a residual image; The refined brightness image is generated based on the residual image and the brightness enhancement image.

[0009] Optionally, the step of processing the refined luminance image through the chroma refinement module in the high dynamic range conversion model to output high dynamic range image data includes: The refined brightness image is projected into a high-dimensional space to obtain a high-dimensional feature map; Local chromaticity correlation features are extracted from the high-dimensional feature map to obtain a local refined map; The high dynamic range image data is obtained by restoring the chromaticity dimension of the local refined image.

[0010] Optionally, the step of calculating prior brightness data based on the original image data through the brightness estimation module in the high dynamic range conversion model includes: The original image data is converted to a linear color space; Based on the converted original image data, the average pixel values ​​in the color channel dimension are taken to obtain the brightness prior data.

[0011] Optionally, the training method for the high dynamic range transformation model includes: Construct an initial high dynamic range conversion model and initialize the parameters of the initial high dynamic range conversion model; Construct a training dataset, which includes original image samples and target image samples; Based on the training dataset, the initial high dynamic range conversion model is iteratively trained with the goal of minimizing the total loss function to obtain the trained high dynamic range conversion model.

[0012] Based on the same inventive concept, a second aspect of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0013] Based on the same inventive concept, a third aspect of this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described above.

[0014] As can be seen from the above, the high dynamic range (HDR) image processing method, electronic device, and storage medium provided in this application include: acquiring original image data; inputting the original image data into a pre-trained HDR conversion model; calculating prior brightness data based on the original image data using a brightness estimation module in the HDR conversion model; and generating brightness features and a brightness-enhanced image based on the prior brightness data and the original image data using the brightness estimation module. The brightness estimation module achieves preliminary dynamic range expansion, initially expanding the compressed dynamic range of the original image data into a broad dynamic range of HDR image data. The brightness estimation module understands the brightness semantics of the original image data, providing key guiding features for subsequent processing. The brightness-enhanced image and the brightness features are processed by a brightness guidance module in the HDR conversion model to obtain a refined brightness image. The multi-scale attention mechanism of the brightness guidance module transforms the preliminary brightness expansion into a high-quality HDR image that conforms to human visual perception and meets professional cinema standards, i.e., a refined brightness image. The refined brightness image is processed by a chroma refinement module in the HDR conversion model to output HDR image data. The color refinement module performs perceptual optimization, wide color gamut adaptation, and color correction and refinement on the refined brightness image to ensure that the final high dynamic range image data meets cinema-grade standards in terms of color accuracy, visual naturalness, and artistic expression. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic flowchart of a high dynamic range image processing method according to an embodiment of this application; Figure 2 This is a schematic diagram of the brightness estimation module according to an embodiment of this application; Figure 3 This is a general flowchart of the high dynamic range image processing method according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the image processing apparatus according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] With the development of the film industry and the rapid advancement of projection technology, high-end cinemas supporting HDR playback are becoming increasingly common. Despite these advancements, the vast majority of existing films are still mastered and distributed in Standard Dynamic Range (SDR) format, thus requiring advanced conversion technologies to fully unleash the potential of High Dynamic Range (HDR) display devices in cinema settings. Existing SDR-to-HDR conversion methods are mostly designed for consumer-grade content such as television, employing global tone mapping strategies that neglect the stringent requirements of cinematic scenes for fine brightness details (such as highlights and shadows) and color accuracy. This results in converted HDR content being prone to detail loss and skin tone distortion in the darkroom environment of a cinema, failing to meet professional cinema projection standards. Moreover, existing training datasets are mostly derived from online videos or demonstration clips, failing to cover the DCDM (Digital Cinema Mastering) workflow of professional film production, and lacking high-fidelity, lossless data support for cinema-grade SDR-HDR frame pairs, thus limiting the applicability of models in cinema-grade content conversion. Furthermore, existing methods lack dedicated optimization mechanisms for cinema-grade wide color gamuts (such as DCI P3 and BT.2020), resulting in insufficient color reproduction accuracy and difficulty in meeting the artistic tone control and wide color gamut presentation requirements of film production.

[0020] From an industry application perspective, the current film market faces a significant technological contradiction: on the one hand, the number of cinemas with HDR playback capabilities is constantly increasing, leading to an increasingly urgent demand for HDR film content; on the other hand, the vast majority of existing films worldwide are still mastered and distributed in SDR format, directly limiting the full potential of HDR cinema equipment. Therefore, SDR-to-HDR conversion technology has become a crucial bridge connecting existing SDR film resources with new HDR projection equipment. Its core objective is to convert low dynamic range and narrow color gamut SDR content into HDR content that meets professional playback standards through algorithmic processing, while preserving the original film's artistic expression and detailed information.

[0021] In terms of technology classification, SDR-to-HDR conversion is an important research direction in the fields of computer vision and digital image processing. Related learning methods can be subdivided into four categories: First, HDR style enhancement, which aims to simulate HDR visual effects on SDR display devices; second, multi-exposure HDR imaging, which generates linear HDR images by fusing multiple SDR images with different exposure parameters; third, single-image HDR reconstruction, which generates linear HDR content based on only a single SDR image; and fourth, SDR TV to HDR TV conversion (SDRTV-to-HDRTV), which outputs HDR TV frames that conform to specific electro-optical conversion functions and wide color gamut RGB primary color standards for TV playback scenarios.

[0022] This application focuses on cinema-grade SDR-to-HDR conversion, which differs fundamentally from SDRTV-to-HDRTV technology due to differences in application scenarios: TV content uses the Rec.709 color gamut and Gamma 2.2 conversion function, and is viewed in diverse home ambient lighting environments, where the requirements for detail reproduction are relatively relaxed; while cinema-grade content uses a wider DCI P3 color gamut and Gamma 2.6 conversion function, is shot with professional cinema cameras, and is projected in darkroom environments, requiring extremely high precision in reproducing fine details such as highlights and shadows, as well as color accuracy. Even slight image distortion can destroy the viewer's immersion.

[0023] This application provides an image processing solution that achieves fine brightness detail restoration and high-fidelity color reproduction by constructing a high-quality dataset and a targeted network architecture, ultimately outputting HDR content that meets professional cinema standards, while also taking into account the HDR conversion needs of online streaming media, filling the gap in existing technologies for cinema-level conversion scenarios.

[0024] The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0025] This application provides a high dynamic range image processing method, referencing... Figure 1 This includes the following steps: Step 101: Obtain the raw image data.

[0026] Specifically, the raw image data is SDR data, conforming to either cinema-grade or streaming standard data. Cinema-grade and streaming standard data differ in their technical parameters. Cinema-grade standard data uses the DCI-P3 color space, gamma 2.6, 16 bits / channel bit depth, and a non-linear XYZ color model. Streaming standard data uses the Rec.709 (sRGB) color space, gamma 2.2, 8 bits / channel bit depth, and a non-linear RGB color model.

[0027] Step 102: Input the original image data into the pre-trained high dynamic range conversion model. The brightness estimation module in the high dynamic range conversion model calculates the brightness prior data based on the original image data. Based on the brightness prior data and the original image data, the brightness estimation module generates brightness features and a brightness-enhanced image.

[0028] Furthermore, the step of calculating prior brightness data based on the original image data through the brightness estimation module in the high dynamic range conversion model includes: The original image data is converted to a linear color space; based on the converted original image data, the average pixel values ​​in the color channel dimension are taken to obtain the prior brightness data.

[0029] Specifically, if the original image data is cinema-grade standard data, it needs to be converted to a linear color space, specifically the linear RGB space, for subsequent data processing and calculations. For example, the converted original image data is... 3840 represents the image width, 2160 represents the image height, and 3 represents the number of channels (corresponding to the R, G, and B color channels). The prior brightness data is calculated using the following formula: (1), Where axis=2 represents the RGB channel dimension, and the average pixel value is taken along the channel dimension to obtain the prior brightness data. By calculating prior brightness data, the perceived brightness of each pixel is approximated, preserving the brightness structure information in the original image data, and compressing 3-channel information into 1 channel, reducing the amount of subsequent calculations.

[0030] The brightness scale block includes a first convolutional layer, a depth-separable convolutional layer, and a second convolutional layer.

[0031] Furthermore, step 102 includes: The original image data and the brightness prior data are concatenated by channels to obtain the concatenated features; The splicing features are input into the first convolutional layer, and the fused features are output through the first convolutional layer. The fused features are input into the depthwise separable convolutional layer, and the brightness features are obtained by feature extraction through the depthwise separable convolutional layer. The brightness features are input into the second convolutional layer, and the brightness scaling map is output through the second convolutional layer; The brightness-enhanced image is generated based on the brightness scaling map and the brightness prior data.

[0032] Specifically, Figure 2 A schematic diagram of the brightness estimation module is shown. Figure 2 The raw image data input to the brightness estimation module This is an SDR frame for theatrical war films. (Based on the original image data.) and brightness characteristics Channel concatenation is performed to obtain concatenated features. Based on these concatenated features, a convolution operation is performed through the first convolutional layer 01 to obtain fused features. For example, the first convolutional layer 01 is a 1×1 convolutional layer. The fused features are input to the depthwise separable convolutional layer 02, and feature extraction is performed through the depthwise separable convolutional layer 02 to obtain the brightness features. , (2), The depthwise separable convolutional layer 02 is a 9×9 convolution. Then, the brightness features are convolved through the second convolutional layer 03 to obtain a scaled brightness map. For example, the second convolutional layer 03 is a 1×1 convolutional layer. Finally, based on the brightness scaling map... and brightness prior data A brightness-enhanced image is generated using the following formula: (3), in, This represents element-wise multiplication. Image enhancement via brightness measurement. Will Figure 2 The brightness of the explosion highlight area in the SDR image is increased.

[0033] Step 103: The brightness enhancement image and the brightness features are processed by the brightness guidance module in the high dynamic range conversion model to obtain a refined brightness image.

[0034] Specifically, the brightness guidance module includes a third convolutional layer, a first brightness guidance attention submodule, a fourth convolutional layer, a second brightness guidance attention submodule, and a fifth convolutional layer, and the multi-scale feature maps include a first feature map, a second feature map, and a third feature map.

[0035] Furthermore, based on the brightness features, the brightness-enhanced image is downsampled at multiple scales by the brightness guidance module to generate a multi-scale feature map, including: The brightness-enhanced image is input into the third convolutional layer to generate the first feature map. Based on the first feature map, it is processed sequentially through the first brightness-guided attention submodule and the fourth convolutional layer to generate the second feature map. Based on the second feature map, it is processed sequentially through the second brightness-guided attention submodule and the fifth convolutional layer to generate the third feature map. The brightness features are applied to the brightness-guided multi-head self-attention submodules in the first and second brightness-guided attention submodules, respectively.

[0036] Figure 3 A general flowchart of the high dynamic range image processing method is shown. Figure 3 The diagram includes three dashed boxes: the left dashed box corresponds to the Luminance Guided Multi-Head Self-Attention (LG-MSA) submodule; the middle dashed box corresponds to the Luminance Guided Attention (LGAB) submodule; and the right dashed box corresponds to the multi-scale feature fusion process. The right dashed box contains multiple Luminance Guided Attention (LGAB) submodules, each with the structure shown in the middle dashed box. The data processing flow of the Luminance Guided Multi-Head Self-Attention (LG-MSA) submodule within each Luminance Guided Attention (LGAB) submodule is shown in the left dashed box.

[0037] After generating brightness features and a brightness-enhanced image through the brightness estimation module, the brightness-enhanced image is input to... Figure 3 In the process shown by the dashed box on the right, the brightness-enhanced image undergoes three levels of downsampling. First, the brightness-enhanced image is input to the third convolutional layer 04. The first feature map is generated by performing a convolution operation on the third convolutional layer 04 using the following formula. : (4), Among them, the third convolutional layer 04 can be Convolutional layers. The first feature map is high-resolution and focuses on image details, used to recover texture and edges. Based on the first feature map, the second feature map is generated by sequentially processing it through the first brightness-guided attention submodule 05 and the fourth convolutional layer 06 using the following formula. : (5), Among them, the first brightness-guided attention submodule 05 includes a single... Figure 3 The dashed box in the middle corresponds to the data processing procedure. The fourth convolutional layer (06) can process... Convolutional layer The stride of the convolutional kernel is 2. The second feature map is of medium resolution, focusing on structure to preserve object shape and local contrast. Based on the second feature map, the third feature map is generated by sequentially processing the second brightness-guided attention submodule 07 and the fifth convolutional layer 08 using the following formula. : (6), Among them, the second brightness-guided attention submodule 07 includes two such... Figure 3 The dashed box in the middle corresponds to the data processing procedure. The fifth convolutional layer 08 can be... Convolutional layers. The third feature map is low-resolution and focuses on semantics, used to understand scene type and global tone. The first, second, and third feature maps have different dimensions, expanding the receptive field by extracting multi-scale features.

[0038] When features pass through the luminance-guided attention submodule LGAB, they are first normalized by a LayerNorm layer, then passed through the luminance-guided multi-head self-attention submodule LG-MSA, residuals are added, and then passed through a LayerNorm layer again. Finally, they pass through an MLP (Multilayer Perceptron) and residuals are added to complete one LGAB calculation. Within each LGAB, the luminance-guided multi-head self-attention (LG-MSA) mechanism modulates attention weights using luminance features, focusing on key bright and dark regions to capture long-range spatial dependencies. During the luminance-guided multi-head self-attention submodule LG-MSA, the third feature map is used... As For example, brightness features Downsampling to the third feature map After corresponding scale, with Reshape into a token sequence It is divided into k=8 attention heads. The feature dimensions of each attention head satisfy the following condition: , For each attention head A cubic linear projection is performed through an unbiased fully connected layer to generate a query (Q), key (K), and value (V): (7), in, This is a learnable projection matrix. The brightness features... Divided into ,modulation And calculate attention: (8), in, A learnable scaling factor. All header outputs are merged and positional encoding is added. Enhanced features are obtained By using brightness guidance, different brightness areas can be optimized. After brightness guidance, details in dark areas are highlighted and enhanced, while highlights are prevented from being over-enhanced, edge contrast is improved, and details are clearer.

[0039] Furthermore, the step of performing feature upsampling and residual fusion on the multi-scale feature map to generate the refined brightness image includes: The multi-scale feature map is upsampled and fused with residuals to generate a residual image; the refined brightness image is generated based on the residual image and the brightness enhancement image.

[0040] Specifically, by upsampling multi-scale feature maps, the resolution of the feature maps is gradually restored, while multi-scale information is fused. Through residual fusion, image details can be preserved, and information can be prevented from decaying layer by layer during the upsampling process.

[0041] Based on the third feature map The fourth feature map is generated by processing the data sequentially through the third brightness-guided attention submodule 09 and the sixth convolutional layer 10 using the following formula. : (9), Among them, the third brightness-guided attention submodule 09 includes two such... Figure 3 The data processing steps corresponding to the dashed box in the middle can be implemented in the sixth convolutional layer 10. Transpose the convolutional layer. Then, apply the fourth feature map. Second feature map The first fused feature is obtained by fusing through the seventh convolutional layer 11. The seventh convolutional layer can be Convolutional layer. Based on the first fused feature. The fifth feature map is generated by processing the data sequentially through the fourth brightness-guided attention submodule 12 and the eighth convolutional layer 13 using the following formula. : (10) Among them, the fourth brightness-guided attention submodule 12 includes two such... Figure 3 The data processing steps corresponding to the dashed box in the middle can be implemented in the eighth convolutional layer 13. Transpose the convolutional layer. Then, apply the fifth feature map. and the first feature map The second fused feature is obtained by fusing through the ninth convolutional layer 14. The ninth convolutional layer 14 can be Convolutional layer. Based on the second fusion feature. The residual image is generated by processing the image sequentially through the fifth brightness-guided attention submodule 15 and the tenth convolutional layer 16 using the following formula. : (11), Among them, the fifth brightness-guided attention submodule 15 includes two such... Figure 3 The data processing steps corresponding to the dashed box in the middle can be implemented in the tenth convolutional layer 16. Convolutional layer. Based on the residual image and the brightness enhancement image, the refined brightness image is generated using the following formula. : (12).

[0042] In this embodiment, a brightness-guided module is used to restore and enhance multi-scale details in the brightness-enhanced image, and a brightness-guided multi-head self-attention mechanism is used to solve the problem of limited local receptive fields in traditional convolution. Global attention coordinates the brightness distribution across the entire image, ensuring that details in dark areas are visible.

[0043] Step 104: Process the refined luminance image through the chroma refinement module in the high dynamic range conversion model to output high dynamic range image data.

[0044] Specifically, based on the refined brightness image generated by the brightness guidance module, the refined brightness image is perceptually optimized and color-corrected and refined through a color refinement model to ensure that the high dynamic range image data meets cinema-grade standards in terms of color accuracy and visual naturalness.

[0045] Furthermore, the step of processing the refined luminance image through the chroma refinement module in the high dynamic range conversion model to output high dynamic range image data includes: The refined brightness image is projected into a high-dimensional space to obtain a high-dimensional feature map; Local chromaticity correlation features are extracted from the high-dimensional feature map to obtain a local refined map; The high dynamic range image data is obtained by restoring the chromaticity dimension of the local refined image.

[0046] Specifically, the refined brightness image is projected into a high-dimensional space using the following formula to obtain a high-dimensional feature map. : (13) in, express Convolutional layer This indicates the convolution kernel, meaning the convolution window size is 1 pixel × 1 pixel. This represents the number of output channels, which is also the number of channels in the features after convolution. This represents the activation function. This represents the Rectified Linear Unit. (Through...) Non-linearity can be introduced into the network, allowing the model to learn complex chromaticity mapping relationships. This high-dimensional mapping provides ample room for subsequent fine-tuning of colors.

[0047] The local refined image is obtained by extracting local chromaticity-related features from the high-dimensional feature map using the following formula: (14) in, Indicates depth Convolutional layer This indicates the number of groups in the grouped convolution, ensuring that each channel segment is processed. convolution. This refers to padding, specifically adding one layer of pixels to the edges of the high-dimensional feature map. The purpose is to make... The spatial dimensions of the feature map after convolution remain consistent with the input. By extracting local chromaticity-related features, color noise is eliminated, ensuring natural color transitions between adjacent pixels and avoiding detail loss due to excessive smoothing.

[0048] The high dynamic range image data is obtained by restoring the chroma dimension of the local refined image using the following formula. : (15).

[0049] Activation function The channel output range is limited to (-1, 1) to facilitate subsequent standardization. A color refinement module achieves perceptual optimization, wide color gamut adaptation, and noise robustness. High dynamic range image data is HDR image data, possessing a wider dynamic range and richer color representation. The color refinement module ensures color accuracy in the HDR space, maintaining color consistency and naturalness.

[0050] Based on steps 101 to 104 above, the high dynamic range (HDR) image processing method provided in this embodiment includes: acquiring original image data; inputting the original image data into a pre-trained HDR conversion model; calculating prior brightness data based on the original image data using a brightness estimation module in the HDR conversion model; and generating brightness features and a brightness-enhanced image based on the prior brightness data and the original image data using the brightness estimation module. The brightness estimation module achieves preliminary dynamic range expansion, initially expanding the compressed dynamic range of the original image data into a broad dynamic range of HDR image data. The brightness estimation module understands the brightness semantics of the original image data, providing key guiding features for subsequent processing. The brightness-enhanced image and the brightness features are processed by a brightness guidance module in the HDR conversion model to obtain a refined brightness image. The multi-scale attention mechanism of the brightness guidance module transforms the preliminary brightness expansion into a high-quality HDR image that conforms to human visual perception and meets professional cinema standards, i.e., a refined brightness image. The refined brightness image is processed by a chroma refinement module in the HDR conversion model to output HDR image data. The color refinement module performs perceptual optimization, wide color gamut adaptation, and color correction and refinement on the refined brightness image to ensure that the final high dynamic range image data meets cinema-grade standards in terms of color accuracy, visual naturalness, and artistic expression.

[0051] In some embodiments, the training method for the high dynamic range conversion model includes: Construct an initial high dynamic range conversion model and initialize the parameters of the initial high dynamic range conversion model; Construct a training dataset, which includes original image samples and target image samples; Based on the training dataset, the initial high dynamic range conversion model is iteratively trained with the goal of minimizing the total loss function to obtain the trained high dynamic range conversion model.

[0052] Specifically, an initial high dynamic range (HDR) conversion model is constructed, comprising an initial luminance estimation module, an initial luminance guidance module, and an initial chroma refinement module. The model structure of the initial HDR conversion model is the same as that of the aforementioned HDR conversion model, and will not be repeated here. The parameters of the initial HDR conversion model are initialized using a Xavier normal distribution initialization strategy. A training dataset is constructed, including both original and target image samples. The training dataset includes the cinematic dataset HDRMovie7K (6775 SDR-HDR frame pairs, DPX format), where SDR image frames are the original image samples and HDR image frames are the target image samples. The training dataset also includes the streaming media dataset HDRMovie1K (984 SDR-HDR frame pairs, PNG format). The training dataset covers various film genres to ensure data diversity.

[0053] The initial high dynamic range conversion model was trained using the training dataset. The total loss function used during training was the L1 loss function and... The weighted sum of the loss functions; the total loss function is the wide color gamut loss function. Among them, The loss function is expressed by the following formula: (16) Where N represents the total number of pixels. This represents the prediction value of the initial high dynamic range conversion model (i.e., the predicted HDR image). This represents the true values ​​in the training dataset (i.e., HDR image frames in the training dataset). and All images are in the CIE 1931 XYZ color space. The formula for converting the predicted HDR image in RGB color space and the HDR image frames in the training dataset into images in the CIE 1931 XYZ color space is as follows: (17) Where M represents the color transformation matrix. During the training process, multiple rounds of iterative training are performed with the objective of minimizing the total loss function. The model parameters of the initial high dynamic range transformation model are updated through back-step iteration to minimize... and The deviation between the two is addressed, and the high dynamic range image data output by the high dynamic range conversion model after training conforms to the Rec.2020 color gamut and PQ transfer function, with a dynamic range of 10,000 nits, improving the color reproduction accuracy and perceptual rationality of HDR images. Figure 3As shown, in the high dynamic range image data output by the high dynamic range conversion model, the explosion highlights are clear, the shadow areas of the soldiers are noise-free, and the colors meet cinema-grade standards.

[0054] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0055] It should be noted that some embodiments of this application have been described above. In some cases, the actions or steps described in the above embodiments can be performed in a different order than that shown in the above embodiments and the desired result can still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0056] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides an image processing apparatus.

[0057] refer to Figure 4 The image processing apparatus includes: Acquisition unit 201 is configured to acquire raw image data; The brightness estimation unit 202 is configured to input the original image data into a pre-trained high dynamic range conversion model, calculate brightness prior data based on the original image data through the brightness estimation module in the high dynamic range conversion model, and generate brightness features and brightness enhancement images based on the brightness prior data and the original image data through the brightness estimation module. The luminance guiding unit 203 is configured to process the luminance enhancement image and the luminance features through the luminance guiding module in the high dynamic range conversion model to obtain a refined luminance image; The chroma refinement unit 204 is configured to process the refined luminance image through the chroma refinement module in the high dynamic range conversion model and output high dynamic range image data.

[0058] In some embodiments, the brightness estimation module includes a first convolutional layer, a depthwise separable convolutional layer, and a second convolutional layer; the brightness estimation unit 202 is configured to perform channel concatenation of the original image data and the brightness prior data to obtain concatenated features; input the concatenated features into the first convolutional layer and output fused features through the first convolutional layer; input the fused features into the depthwise separable convolutional layer and perform feature extraction through the depthwise separable convolutional layer to obtain the brightness features; input the brightness features into the second convolutional layer and output a brightness scaling map through the second convolutional layer; and generate the brightness-enhanced image based on the brightness scaling map and the brightness prior data.

[0059] In some embodiments, the brightness guidance unit 203 is configured to perform multi-scale feature downsampling on the brightness enhancement image based on the brightness features through the brightness guidance module to generate a multi-scale feature map; and to perform feature upsampling and residual fusion on the multi-scale feature map to generate the refined brightness image.

[0060] In some embodiments, the brightness guidance module includes a third convolutional layer, a first brightness guidance attention submodule, a fourth convolutional layer, a second brightness guidance attention submodule, and a fifth convolutional layer; the multi-scale feature map includes a first feature map, a second feature map, and a third feature map; the brightness guidance unit 203 is configured to input the brightness enhancement image into the third convolutional layer, generate the first feature map through the third convolutional layer; process the first feature map sequentially through the first brightness guidance attention submodule and the fourth convolutional layer to generate the second feature map; process the second feature map sequentially through the second brightness guidance attention submodule and the fifth convolutional layer to generate the third feature map; wherein the brightness features are respectively applied to the brightness guidance multi-head self-attention submodules in the first brightness guidance attention submodule and the second brightness guidance attention submodule.

[0061] In some embodiments, the brightness guidance unit 203 is configured to perform feature upsampling and residual fusion on the multi-scale feature map to generate a residual image; and generate the refined brightness image based on the residual image and the brightness enhancement image.

[0062] In some embodiments, the chroma refinement unit 204 is configured to project the refined luminance image into a high-dimensional space to obtain a high-dimensional feature map; extract local chroma-related features from the high-dimensional feature map to obtain a local refined map; and restore the chroma dimension of the local refined map to obtain the high dynamic range image data.

[0063] In some embodiments, the brightness estimation unit 202 is configured to convert the original image data to a linear color space; and based on the converted original image data, to take the average of the pixel values ​​in the color channel dimension to obtain the brightness prior data.

[0064] In some embodiments, a model training unit is further included, configured to construct an initial high dynamic range conversion model and initialize the parameters of the initial high dynamic range conversion model; construct a training dataset, which includes original image samples and target image samples; and iteratively train the initial high dynamic range conversion model based on the training dataset with the objective of minimizing the total loss function to obtain a trained high dynamic range conversion model.

[0065] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0066] The apparatus described above is used to implement the corresponding high dynamic range image processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0067] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the high dynamic range image processing method described in any of the above embodiments.

[0068] Figure 5 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0069] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0070] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0071] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0072] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0073] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0074] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0075] The electronic devices described above are used to implement the corresponding high dynamic range image processing methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0076] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the high dynamic range image processing method as described in any of the above embodiments.

[0077] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0078] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the high dynamic range image processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0079] Based on the same concept, corresponding to any of the above embodiments, this application also provides a computer program product, including computer program instructions, which, when run on a computer, cause the computer to perform the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0080] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0081] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0082] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0083] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A high dynamic range image processing method, characterized in that, include: Obtain the raw image data; The original image data is input into a pre-trained high dynamic range conversion model. The brightness estimation module in the high dynamic range conversion model calculates the brightness prior data based on the original image data. Based on the brightness prior data and the original image data, the brightness estimation module generates brightness features and a brightness-enhanced image. The brightness enhancement image and the brightness features are processed by the brightness guidance module in the high dynamic range conversion model to obtain a refined brightness image; The refined luminance image is processed by the chroma refinement module in the high dynamic range conversion model to output high dynamic range image data.

2. The method according to claim 1, characterized in that, The brightness estimation module includes a first convolutional layer, a depth-separable convolutional layer, and a second convolutional layer; The step of generating brightness features and a brightness-enhanced image through the brightness estimation module based on the prior brightness data and the original image data includes: The original image data and the brightness prior data are concatenated by channels to obtain the concatenated features; The splicing features are input into the first convolutional layer, and the fused features are output through the first convolutional layer. The fused features are input into the depthwise separable convolutional layer, and the brightness features are obtained by feature extraction through the depthwise separable convolutional layer. The brightness features are input into the second convolutional layer, and the brightness scaling map is output through the second convolutional layer; The brightness-enhanced image is generated based on the brightness scaling map and the brightness prior data.

3. The method according to claim 1, characterized in that, The process of processing the brightness enhancement image and the brightness features through the brightness guidance module in the high dynamic range conversion model to obtain a refined brightness image includes: Based on the brightness features, the brightness enhancement image is downsampled at multiple scales by the brightness guidance module to generate a multi-scale feature map. The multi-scale feature map is upsampled and the residual is fused to generate the refined brightness image.

4. The method according to claim 3, characterized in that, The brightness guidance module includes a third convolutional layer, a first brightness guidance attention submodule, a fourth convolutional layer, a second brightness guidance attention submodule, and a fifth convolutional layer; the multi-scale feature map includes a first feature map, a second feature map, and a third feature map; The step of performing multi-scale feature downsampling on the brightness-enhanced image based on the brightness features, through the brightness guidance module, to generate a multi-scale feature map includes: The brightness-enhanced image is input into the third convolutional layer, and the first feature map is generated through the third convolutional layer; The first feature map is processed sequentially through the first brightness-guided attention submodule and the fourth convolutional layer to generate the second feature map; The second feature map is processed sequentially through the second brightness-guided attention submodule and the fifth convolutional layer to generate the third feature map; wherein the brightness features are applied to the brightness-guided multi-head self-attention submodules in the first brightness-guided attention submodule and the second brightness-guided attention submodule, respectively.

5. The method according to claim 3, characterized in that, The step of upsampling and residual fusion of the multi-scale feature map to generate the refined brightness image includes: The multi-scale feature map is upsampled and fused with residuals to generate a residual image; The refined brightness image is generated based on the residual image and the brightness enhancement image.

6. The method according to claim 1, characterized in that, The process of processing the refined luminance image through the chroma refinement module in the high dynamic range conversion model to output high dynamic range image data includes: The refined brightness image is projected into a high-dimensional space to obtain a high-dimensional feature map; Local chromaticity correlation features are extracted from the high-dimensional feature map to obtain a local refined map; The high dynamic range image data is obtained by restoring the chromaticity dimension of the local refined image.

7. The method according to claim 1, characterized in that, The brightness prior data calculated based on the original image data by the brightness estimation module in the high dynamic range conversion model includes: The original image data is converted to a linear color space; Based on the converted original image data, the average pixel values ​​in the color channel dimension are taken to obtain the brightness prior data.

8. The method according to claim 1, characterized in that, The training method for the high dynamic range conversion model includes: Construct an initial high dynamic range conversion model and initialize the parameters of the initial high dynamic range conversion model; Construct a training dataset, which includes original image samples and target image samples; Based on the training dataset, the initial high dynamic range conversion model is iteratively trained with the goal of minimizing the total loss function to obtain the trained high dynamic range conversion model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.

Images