Model training method and device, tone mapping method and device, electronic device, computer storage medium, and computer program product

Abstract

Embodiments of the present application provide a model training method and device, a tone mapping method and device, electronic equipment, a computer storage medium, and a computer program product. The model training method comprises: obtaining training data, the training data comprising a tone image sample and multi-scale visual structure data of the tone image sample; determining a diffusion model comprising a pre-training module, a pre-configured analysis module and an encoding module; taking the tone image sample as the main input and output of the diffusion model, taking the multi-scale visual structure data as the conditional input of the pre-configured analysis module, and iteratively training at least the network parameters of the encoding module to obtain a tone mapping model. The pre-configured analysis module analyzes the context feature map of the tone image sample and reconstructs the context feature map using the multi-scale visual structure data as the reconstruction condition, and reconstructs the visual structure features of the analysis result to obtain a reconstructed feature map, and the encoding module encodes the reconstructed feature map in the latent space to obtain the tone image sample.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training method and apparatus, a tone mapping method and apparatus, an electronic device, a computer storage medium, and a computer program product. Background Technology

[0002] With the increasing use of tonal images, such as Dynamic Range Imaging (DR) images, tone mapping, a technique for converting high-tone images, such as HDR images, into low-tone images, such as Low Dynamic Range Imaging (LDR) images, for display, plays an increasingly important role in real-world visual systems. However, obtaining paired tonal image samples (e.g., HDR image samples and high-quality LDR image samples) is difficult, posing a challenge to deep learning-based tone mapping methods. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a model training method and apparatus, a tone mapping method and apparatus, an electronic device, a computer storage medium, and a computer program product to solve the above problems.

[0004] According to a first aspect of the present invention, a model training method is provided, comprising: acquiring training data, the training data including tone image samples and multi-scale visual structure data of the tone image samples; determining a diffusion model, the diffusion model including a pre-training module, a pre-configured parsing module, and an encoding module; using the tone image samples as the main input and output of the diffusion model, and using the multi-scale visual structure data as the conditional input of the pre-configured parsing module, iteratively training at least the network parameters of the encoding module to obtain a tone mapping model; wherein, the pre-training module is used to generate a context feature map based on the random noise addition result of the tone image samples, the pre-configured parsing module is used to perform visual structure feature parsing on the context feature map and using the multi-scale visual structure data as a reconstruction condition, performing visual structure feature reconstruction on the parsing result to obtain a reconstructed feature map, and the encoding module is used to perform latent space encoding on the reconstructed feature map to obtain the tone image samples.

[0005] In another implementation of the present invention, at least the network parameters of the encoding module are iteratively trained, including: the pre-configured parsing module determines the visual structural features of the initial iteration as the shared visual structural features for each iteration; in each iteration training, the pre-configured parsing module performs visual structural feature parsing on the context feature map of the current iteration to generate the hue features and visual structural features of the current iteration; the pre-configured parsing module uses the multi-scale visual structural data as a reconstruction condition to fuse the hue features of the current iteration with the shared visual structural features to obtain a reconstructed feature map.

[0006] In another implementation of the present invention, the pre-training module includes a generator network and a control network, wherein at least the network parameters of the encoding module are iteratively trained, including: while iteratively training the network parameters of the encoding module, at least the multi-scale visual structure data is used as a conditional input to the control network to fine-tune the control network.

[0007] In another implementation of the present invention, the training data further includes the blurred brightness data of the tone image samples, wherein at least the multi-scale visual structure data is used as the conditional input of the control network to fine-tune the control network, including: using the multi-scale visual structure data and the blurred brightness data as the first conditional input and the second conditional input of the control network, respectively, to fine-tune the control network.

[0008] In another implementation of the present invention, the control network is fine-tuned by using the multi-scale visual structure data and the blurred brightness data as the first conditional input and the second conditional input, respectively. This includes feeding the visual structure features as the first conditional input into the first encoding layer of the control network and feeding the blurred brightness data as the second conditional input into the second encoding layer of the control network, thereby fine-tuning the control network. In each encoding layer of the control network, the first encoding layer is located on the input side of the second encoding layer.

[0009] In another implementation of the present invention, each decoding layer of the control network is fused into the corresponding decoding layer of the generation network.

[0010] According to a second aspect of the present invention, a tone mapping method is provided, comprising: obtaining a tone mapping model, the tone mapping model including a pre-training module, a pre-configured parsing module, and an encoding module, the tone mapping model being trained by the method described in the first aspect; performing visual structure processing on a high-tone image to obtain multi-scale visual structure data of the high-tone image; feeding at least the high-tone image into the main input of the tone mapping model, and feeding the multi-scale visual structure data into the conditional input of the pre-configured parsing module, and outputting a low-tone image from the tone mapping model.

[0011] In another implementation of the present invention, the method further includes: performing brightness processing on the high-tone image to obtain blurred brightness data of the high-tone image, wherein, when outputting a low-tone image from the tone mapping model, the multi-scale visual structure data and the blurred brightness data are fed into a first conditional input and a second conditional input of the control network in the pre-trained module.

[0012] According to a third aspect of the present invention, a model training apparatus is provided, comprising: an acquisition unit for acquiring training data, the training data including tone image samples and multi-scale visual structure data of the tone image samples; a determination unit for determining a diffusion model, the diffusion model including a pre-training module, a pre-configuration parsing module, and an encoding module; and a training unit for iteratively training at least the network parameters of the encoding module using the tone image samples as the main input and output of the diffusion model, and using the multi-scale visual structure data as the conditional input of the pre-configuration parsing module to obtain a tone mapping model, wherein the pre-training module is used to generate a context feature map based on the random noise addition result of the tone image samples, the pre-configuration parsing module is used to perform visual structure feature parsing on the context feature map and using the multi-scale visual structure data as a reconstruction condition to perform visual structure feature reconstruction on the parsing result to obtain a reconstructed feature map, and the encoding module is used to perform latent space encoding on the reconstructed feature map to obtain the tone image samples.

[0013] According to a fourth aspect of the present invention, a tone mapping apparatus is provided, comprising: a model acquisition unit for acquiring a tone mapping model, the tone mapping model including a pre-training module, a pre-configured parsing module, and an encoding module, the tone mapping model being trained by the method described in the first aspect; an image processing unit for performing visual structure processing on a high-tone image to obtain multi-scale visual structure data of the high-tone image; and a tone mapping unit for feeding at least the high-tone image into the main input of the tone mapping model, and feeding the multi-scale visual structure data into the conditional input of the pre-configured parsing module, and outputting a low-tone image from the tone mapping model.

[0014] According to a fifth aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, the executable instruction causing the processor to perform the method as described in the first aspect.

[0015] According to a sixth aspect of the present invention, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0016] According to a seventh aspect of the present invention, a computer program product is provided, including computer instructions that, when executed by a processor, implement the method described in the first aspect.

[0017] In the embodiment of the present invention, visual structural features can effectively characterize latent features that are independent of the dynamic tonal range. The pre-configured parsing module is used to parse the context feature map for visual structural features and use multi-scale visual structural data as reconstruction conditions. The parsing results are then used to reconstruct the visual structural features to obtain the reconstructed feature map. In other words, the multi-scale visual structural data of the tonal image samples enables the pre-configured parsing module to reconstruct the parsing results of the visual structural features, so that the network parameters of the encoding module can learn the latent representation of the visual structural features. Therefore, while ensuring the tonal mapping effect of the trained tonal mapping model, the requirements for the dynamic tonal range of the tonal image samples are reduced, that is, zero-time learning tonal mapping is achieved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a flowchart illustrating the steps of a model training method according to some embodiments of the present invention.

[0020] Figure 2 This is a schematic structural diagram of the tone mapping model and diffusion model in an embodiment of the present invention.

[0021] Figure 3 This is a schematic structural diagram of a pre-training module according to some embodiments of the present invention.

[0022] Figure 4 This is a flowchart illustrating the steps of a tone mapping method according to another embodiment of the present invention.

[0023] Figure 5 This is a structural block diagram of a model training device according to other embodiments of the present invention.

[0024] Figure 6 This is a structural block diagram of a tone mapping device according to other embodiments of the present invention.

[0025] Figure 7 This is a schematic diagram of the structure of an electronic device according to other embodiments of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0027] The specific implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0028] Tone mapping is a long-standing computer vision task focused on imaging HDR scenes by converting HDR images into high-quality LDR images. Unlike LDR images, HDR images typically have a wider bit width, resulting in greater diversity in pixel value distribution, especially in extreme conditions such as backlit scenes, where they can capture more information. Although the human eye can perceive both HDR and LDR scenes, typical displays struggle to accurately represent the rich information in HDR images. Therefore, mapping HDR information to an LDR space that retains both a natural appearance and accurate structure is a crucial task for real-world visual systems.

[0029] The visual structural features of an image typically refer to features in an image other than brightness or color information. They describe the visual structural features of objects in the image, such as shape, texture, edges, and contours. For example, shape is the basic outline or boundary of an object in an image; it can be a simple geometric shape or a complex natural shape. Texture, for another example, is the pixel grayscale or color distribution pattern in a local area of ​​an image, describing the detail and roughness of an object's surface. Edges, for yet another example, are locations in an image where brightness or color changes significantly, usually corresponding to the boundaries or contours of an object. Contours, for yet another example, are the outer boundaries of an object in an image, describing the overall shape and size of the object.

[0030] In tone mapping tasks, preserving the visual structural features of an image is particularly important. Visual structural features ensure the preservation of image structural details and features. The remaining "tone" refers to the distribution of brightness values ​​in the image, which represents the different brightness values ​​of different regions within the image.

[0031] Figure 1 This is a flowchart illustrating the steps of a model training method according to some embodiments of the present invention. The scheme of this embodiment can be applied to any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, tablets, etc.), and PCs. For example, in the model training phase, a computing device (e.g., a data center) configured with a CPU (processor unit) + GPU (acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. The computing device, such as a data center, can be deployed in a cloud server such as a private cloud, a hybrid cloud, or a dedicated cloud. Correspondingly, in the inference phase, a computing device configured with a CPU (processor unit) + GPU (acceleration unit) architecture (e.g., a terminal device or a cloud server) can also be used for inference operations. Specifically, Figure 1 Model training methods include:

[0032] S110: Acquire training data, which includes tonal image samples and multi-scale visual structure data of tonal image samples.

[0033] It should be understood that tonal image samples can be LRD image samples, such as high-quality LRD image samples. Multi-scale visual structure data can be MSCN data obtained by subtracting the mean and normalizing the contrast (MSCN) of tonal image samples. In MSCN processing, the luminance component of the image is removed (i.e., mean subtraction) to allow focus on the structural information of the image. Generally, luminance information is usually related to external factors such as image lighting conditions and exposure time, while visual structure information reflects the inherent properties of objects in the image. MSCN data is beneficial for extracting and analyzing the visual structure information of images, thereby improving the accuracy and efficiency of image processing.

[0034] S120: Determine the diffusion model, which includes a pre-training module, a pre-configured parsing module, and an encoding module.

[0035] It should be understood that a diffusion model (DM) can be a generative diffusion model or other diffusion models. A diffusion model is a type of generative model whose purpose is to learn to generate data, given training data, so as to generate diverse outputs similar to the training data.

[0036] S130: Using tone image samples as the main input and output of the diffusion model, and multi-scale visual structure data as the conditional input of the pre-configured parsing module, the network parameters of the encoding module are iteratively trained at least to obtain the tone mapping model. The pre-training module is used to generate a context feature map based on the random noise addition result of the tone image samples. The pre-configured parsing module is used to perform visual structure feature parsing on the context feature map, and uses multi-scale visual structure data as the reconstruction condition to perform visual structure feature reconstruction on the parsing result to obtain the reconstructed feature map. The encoding module is used to perform latent space encoding on the reconstructed feature map to obtain tone image samples.

[0037] It should be understood that the diffusion model enables iterative diffusion processes, starting with random noise and gradually refining it through multiple steps until an output image is obtained. In each iteration, the expanded model is able to estimate how to denoise from the current input. Because only small changes are made in each iteration, any errors in the estimation from earlier iterations can be corrected in the updates of later iterations.

[0038] In the embodiment of the present invention, visual structural features can effectively characterize latent features that are independent of the dynamic tonal range. The pre-configured parsing module is used to parse the context feature map for visual structural features and use multi-scale visual structural data as reconstruction conditions. The parsing results are then used to reconstruct the visual structural features to obtain the reconstructed feature map. In other words, the multi-scale visual structural data of the tonal image samples enables the pre-configured parsing module to reconstruct the parsing results of the visual structural features, so that the network parameters of the encoding module can learn the latent representation of the visual structural features. Therefore, while ensuring the tonal mapping effect of the trained tonal mapping model, the requirements for the dynamic tonal range of the tonal image samples are reduced, that is, zero-time learning tonal mapping is achieved.

[0039] In some specific examples, such as Figure 2As shown, the diffusion model 1000 includes a pre-training module 10, a pre-configuration parsing module 20, and an encoding module 30. In the model training method, the diffusion model 1000 is trained iteratively multiple times. The random noise-adding result of the hue image samples can be used as the main input of the pre-training module 10. The output of the pre-training module 10 is connected to the input of the pre-configuration parsing module 20, and the output of the pre-configuration parsing module 20 is connected to the input of the encoding module 30. The output of the encoding module 30 is used as the output of the diffusion model 1000. The pre-training module 10 may include its own encoder and decoder. In other examples, the luminance processing unit 40 is used to process the hue image samples or hyper-hue images, correspondingly obtaining blurred luminance data such as low-channel luminance data; the visual structure processing unit 50 is used to process the hue image samples or hyper-hue images, correspondingly obtaining visual structure data such as MSCN data.

[0040] In some examples, the network parameters of the pre-trained module 10 can be pre-trained. During multiple iterations of training the diffusion model 1000, the pre-configuration logic of the pre-configured parsing module 20 is maintained, and the network parameters of the encoding module 30 are adjusted.

[0041] It should be understood that the pre-configuration logic of the pre-configuration parsing module 20 includes parsing logic 21 and reconstruction logic 22. For example, the parsing logic can be referred to as Tone Structure Decomposition (TSD), and the reconstruction logic can be referred to as Tone Structure Reconstruction (TSR).

[0042] Specifically, the pre-configured parsing module performs visual structural feature parsing on the context feature map of each iteration, generating the hue features and visual structural features for that iteration. For example, in each iteration, the parsing logic performs visual structural feature parsing on the context feature map of that iteration, obtaining the hue features and visual structural features of that iteration as the parsing results. Furthermore, the pre-configured parsing module determines the visual structural features of the initial iteration as the shared visual structural features for each iteration. Correspondingly, the reconstruction logic uses multi-scale visual structural data as reconstruction conditions to fuse the hue features of that iteration with the shared visual structural features to obtain the reconstructed feature map. Because the backsampling process of the diffusion model is modified, and visual structural features are significantly integrated into the intermediate results of the extended model, the hue mapping effect of the hue mapping model is significantly guaranteed.

[0043] It should be understood that the parsing logic performs visual structural feature analysis on the context feature map of this iteration, obtaining the hue feature and visual structural feature of this iteration as the parsing result. For example, the hue feature of this iteration can be obtained by filtering the context feature map of this iteration with a linear filter kernel to obtain a local average feature map. In addition, the context feature map can also be filtered with a linear filter kernel to obtain a local annotation difference feature map. The linear filter can be a Gaussian filter kernel. More specifically, the local average feature map and the local annotation difference feature map can be implemented using the following formulas 1 and 2, respectively:

[0044]

[0045]

[0046] Where I(x,y) represents the context feature map; x, y represent the pixel positions of the context feature map; K is the size of the linear filter kernel, i, j represent the offset of the linear filter kernel with respect to position (x,y); μ(i,j) represents the local average feature map (i.e., the local average value corresponding to each pixel) obtained by filtering the context feature map with the linear filter kernel; σ(i,j) represents the local annotation difference feature map (i.e., the local annotation difference corresponding to each pixel) obtained by filtering the context feature map with the linear filter kernel; ω(x,y) represents the weight value of the linear filter kernel, such as the Gaussian filter kernel, at position (x,y).

[0047] Furthermore, the visual structural features of this iteration, for example, can be characterized as the proportional relationship between the degree to which the pixel value of each pixel in the feature map deviates from the local average and the local annotation difference, in order to efficiently and accurately represent the visual structural features in the feature map below. More specifically, the visual structural features can be implemented using Equation 3:

[0048]

[0049] Right now, It represents visual structural features; ε is an empirical value, for example, ε can be a constant.

[0050] It should also be understood that the reconstruction logic uses multi-scale visual structure data as reconstruction conditions to fuse the tonal features of the current iteration with the shared visual structure features to obtain a reconstructed feature map. For example, the tonal features of the current iteration and the shared visual structure features can be weighted based on the noise variation parameters of the diffusion model to obtain the reconstructed feature map. More specifically, the reconstruction logic can be implemented using the following formula:

[0051]

[0052] Among them, I predRepresents the reconstructed feature map; Characterize shared visual structural features, such as the visual structural features of the first iteration of a diffusion model; The visual structural features are represented by μ; μ represents the local average feature map obtained by filtering the context feature map using a linear filter kernel; σ represents the local annotation difference feature map obtained by filtering the context feature map using a linear filter kernel; γ represents the parameter representing the noise variation (e.g., increase) during the diffusion process of the diffusion model.

[0053] In other examples, the network parameters of the control network in the pre-training module 10 are fine-tuned during multiple iterations of training the diffusion model 1000. For example, the pre-training module 10 includes a generator network 11 and a control network 12. As an example of iteratively training at least the network parameters of the encoding module, the control network can be fine-tuned simultaneously with multi-scale visual structural data as conditional input. This example can also be referred to as fine-scale control, where multi-scale visual structural data, such as MSCN, is used as conditional input. For example, it can be input into the first encoding layer of the control network, and correspondingly, intermediate results at different scales can be added to the feature maps of the corresponding decoder layers of the generator branch to implicitly control the content of the generated feature maps.

[0054] More specifically, the training data also includes blurred luminance data from tonal image samples. For example, blurred luminance data can be obtained by processing YUV color-coded images. In the YUV color coding system, "Y" represents luminance, while "U" and "V" represent chrominance, used to describe image color and saturation. For instance, blurred luminance data can be obtained by blurring the Y channel (luminance map) of a YUV color-coded tonal image sample using a Gaussian filter. During both training and inference, the luminance guidance is a low-pass luminance map. The network parameters use information contained in the MSCN data to generate fine-scale textures and generate coarser-scale content by utilizing the blurred luminance map as a low-pass luminance map.

[0055] Specifically, as an example of fine-tuning the control network by using at least multi-scale visual structure data as conditional inputs, multi-scale visual structure data and blurred brightness data can be used as the first and second conditional inputs to the control network, respectively, for fine-tuning. For example, as... Figure 3As shown, the decoding layers of the control network 12 are fused to the corresponding decoding layers of the generator network 11. In some examples, the decoding layers of the control network 12 can be zero-convolutional layers. Furthermore, in the coding layers of the control network, the first coding layer is located on the input side of the second coding layer; the first coding layer receives a first conditional input, and the second coding layer receives a second conditional input.

[0056] In other words, as an example of fine-tuning the control network, visual structural features can be fed into the first encoding layer of the control network as the first conditional input, and blurred brightness data can be fed into the second encoding layer of the control network as the second conditional input, thereby fine-tuning the control network.

[0057] This example could also be referred to as coarse-scale control, as multi-scale visual structure data such as those from MSCN significantly eliminates brightness information, and brightness may also be distorted. For example, the relative brightness intensity between different parts of an image may be incorrect, resulting in an unnatural appearance. Therefore, another conditional input that includes brightness information is needed.

[0058] The following will combine Figure 4 To describe another embodiment of the tone mapping method of the present invention. Specifically, Figure 4 Tone mapping methods include:

[0059] S410: Obtain the tone mapping model, which includes a pre-trained module, a pre-configured parsing module, and an encoding module.

[0060] It should be understood that the tone mapping model is trained using the model training method described above.

[0061] S420: Perform visual structure processing on the high-tone image to obtain multi-scale visual structure data of the high-tone image.

[0062] It should be understood that multi-scale visual structure data can be MSCN data obtained by subtracting the mean and normalizing the contrast (MSCN) of high-tone image samples. In MSCN processing, the luminance component of the image is removed (i.e., mean subtraction) to allow focus on the structural information of the image. Generally, luminance information is usually related to external factors such as image lighting conditions and exposure time, while visual structure information reflects the inherent properties of objects in the image. MSCN data is beneficial for extracting and analyzing the visual structure information of images, thereby improving the accuracy and efficiency of image processing.

[0063] S430: Feed at least the high-tone image into the main input of the tone mapping model and feed multi-scale visual structure data into the conditional input of the pre-configured parsing module, outputting a low-tone image from the tone mapping model.

[0064] It should be understood that examples of high-tone images are HDR images, and examples of low-tone images are LDR images.

[0065] In the embodiment of this invention, visual structural features can effectively characterize latent features independent of dynamic tonal range. A pre-configured parsing module performs visual structural feature parsing on the context feature map and uses multi-scale visual structural data as reconstruction conditions. The parsing results are then used to reconstruct the visual structural features to obtain a reconstructed feature map. In other words, the multi-scale visual structural data of the tonal image samples enables the pre-configured parsing module to reconstruct the parsing results of the visual structural features, allowing the network parameters of the encoding module to learn the latent representations of visual structural features. Therefore, while ensuring the tonal mapping effect of the trained tonal mapping model, the requirements for the dynamic tonal range of the tonal image samples are reduced. Thus, the tonal mapping model can perform tonal mapping on high-tone images to obtain low-tone images.

[0066] In some examples, the diffusion model 1000 includes a pre-training module 10, a pre-configured parsing module 20, and an encoding module 30. In the model training method, the diffusion model 1000 is trained iteratively multiple times. Randomly noise-added results of tone image samples can serve as the main input to the pre-training module 10. The output of the pre-training module 10 is connected to the input of the pre-configured parsing module 20, and the output of the pre-configured parsing module 20 is connected to the input of the encoding module 30. The output of the encoding module 30 serves as the output of the diffusion model 1000. It should be understood that the pre-configuration logic of the pre-configured parsing module 20 includes parsing logic 21 and reconstruction logic 22. Accordingly, during the inference process of the tone mapping model, multi-scale visual structural data is fed into the conditional input of the reconstruction logic 22.

[0067] In other examples, the high-tone image is subjected to luminance processing to obtain blurred luminance data of the high-tone image. The blurred luminance data of the high-tone image can be obtained by blurring the Y channel (luminance map) of the YUV color-coded high-tone image using a Gaussian filter. During multiple iterations of training the diffusion model 1000, the network parameters of the control network in the pre-training module 10 are fine-tuned. For example, the pre-training module 10 includes a generator network 11 and a control network 12. While iteratively training the network parameters of the encoding module, the control network can be fine-tuned by using at least multi-scale visual structure data as conditional inputs. Accordingly, during the inference process of the tone mapping model, when outputting a low-tone image from the tone mapping model, the multi-scale visual structure data and blurred luminance data are fed into the first and second conditional inputs of the control network in the pre-training module.

[0068] The following will combine Figure 5 The present invention describes a model training apparatus according to other embodiments of the invention. The scheme of this embodiment can be applied to any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, tablets, etc.), and PCs. For example, in the model training phase, a computing device (e.g., a data center) configured with a CPU (processor unit) + GPU (acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. The computing device, such as a data center, can be deployed in a cloud server such as a private cloud, a hybrid cloud, or a dedicated cloud. Correspondingly, in the inference phase, a computing device configured with a CPU (processor unit) + GPU (acceleration unit) architecture (e.g., a terminal device or a cloud server) can also be used for inference operations. Specifically, Figure 5 The model training apparatus includes:

[0069] The acquisition unit 510 acquires training data, which includes tone image samples and multi-scale visual structure data of the tone image samples.

[0070] Unit 520 is determined to determine the diffusion model, which includes a pre-training module, a pre-configuration parsing module, and an encoding module.

[0071] Training unit 530 uses the hue image samples as the main input and output of the diffusion model, and the multi-scale visual structure data as the conditional input of the pre-configured parsing module. Iteratively trains the network parameters of the encoding module at least to obtain a hue mapping model. The pre-training module is used to generate a context feature map based on the random noise addition result of the hue image samples. The pre-configured parsing module is used to perform visual structure feature parsing on the context feature map and use the multi-scale visual structure data as the reconstruction condition to perform visual structure feature reconstruction on the parsing result to obtain a reconstructed feature map. The encoding module is used to perform latent space encoding on the reconstructed feature map to obtain the hue image samples.

[0072] In the embodiment of the present invention, visual structural features can effectively characterize latent features that are independent of the dynamic tonal range. The pre-configured parsing module is used to parse the context feature map for visual structural features and use multi-scale visual structural data as reconstruction conditions. The parsing results are then used to reconstruct the visual structural features to obtain the reconstructed feature map. In other words, the multi-scale visual structural data of the tonal image samples enables the pre-configured parsing module to reconstruct the parsing results of the visual structural features, so that the network parameters of the encoding module can learn the latent representation of the visual structural features. Therefore, while ensuring the tonal mapping effect of the trained tonal mapping model, the requirements for the dynamic tonal range of the tonal image samples are reduced, that is, zero-time learning tonal mapping is achieved.

[0073] In other embodiments, the training unit is specifically used for: the pre-configured parsing module determining the visual structural features of the initial iteration as the shared visual structural features for each iteration; in each iteration of training, the pre-configured parsing module performs visual structural feature parsing on the context feature map of the current iteration to generate the hue features and visual structural features of the current iteration; the pre-configured parsing module uses the multi-scale visual structural data as reconstruction conditions to fuse the hue features of the current iteration with the shared visual structural features to obtain a reconstructed feature map.

[0074] In other embodiments, the pre-training module includes a generator network and a control network. The training unit is specifically configured to: while iteratively training the network parameters of the encoding module, fine-tune the control network using at least the multi-scale visual structure data as conditional input.

[0075] In other embodiments, the training data further includes blurred brightness data of the tonal image samples. The training unit is specifically used to fine-tune the control network by using the multi-scale visual structure data and the blurred brightness data as the first and second conditional inputs, respectively.

[0076] In other embodiments, the training unit is specifically used to: feed the visual structural features as a first conditional input into a first encoding layer of the control network, and feed the blurred brightness data as a second conditional input into a second encoding layer of the control network, to fine-tune the control network, wherein, in each encoding layer of the control network, the first encoding layer is located on the input side of the second encoding layer.

[0077] In other embodiments, the various decoding layers of the control network are merged into the corresponding decoding layers of the generation network.

[0078] The following will combine Figure 6 Structural block diagrams illustrating tone mapping apparatuses according to other embodiments of the present invention are provided. Specifically, Figure 6 The tone mapping device includes:

[0079] The model acquisition unit 610 acquires a tone mapping model, which includes a pre-training module, a pre-configuration parsing module, and an encoding module.

[0080] Image processing unit 620 performs visual structure processing on high-tone image to obtain multi-scale visual structure data of the high-tone image;

[0081] The tone mapping unit 630 feeds at least the high-tone image into the main input of the tone mapping model and feeds the multi-scale visual structure data into the conditional input of the pre-configured parsing module, and outputs a low-tone image from the tone mapping model.

[0082] In the embodiment of this invention, visual structural features can effectively characterize latent features independent of dynamic tonal range. A pre-configured parsing module performs visual structural feature parsing on the context feature map and uses multi-scale visual structural data as reconstruction conditions. The parsing results are then used to reconstruct the visual structural features to obtain a reconstructed feature map. In other words, the multi-scale visual structural data of the tonal image samples enables the pre-configured parsing module to reconstruct the parsing results of the visual structural features, allowing the network parameters of the encoding module to learn the latent representations of visual structural features. Therefore, while ensuring the tonal mapping effect of the trained tonal mapping model, the requirements for the dynamic tonal range of the tonal image samples are reduced. Thus, the tonal mapping model can perform tonal mapping on high-tone images to obtain low-tone images.

[0083] In other embodiments, examples of the image processing unit 620 are as follows: Figure 2The image processing unit 620 (i.e., the brightness processing unit 40) is specifically used to: perform brightness processing on the high-tone image to obtain blurred brightness data of the high-tone image. The tone mapping unit 630 is specifically used to: when outputting a low-tone image from the tone mapping model, feed the multi-scale visual structure data and the blurred brightness data into the first conditional input and the second conditional input of the control network in the pre-trained module.

[0084] The specific implementation of each module in the apparatus for implementing the above methods can be found in the descriptions of the corresponding steps in the above method embodiments, which also have corresponding beneficial effects, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the apparatus and modules for implementing the above methods can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0085] Reference Figure 7 The diagram shows a structural schematic of an electronic device according to another embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0086] like Figure 7 As shown, the electronic device may include: a processor 702 for executing program 710, a communications interface 704, a memory 706, and a communications bus 708.

[0087] The processor, communication interface, and memory communicate with each other via a communication bus.

[0088] A communication interface is used to communicate with other electronic devices or servers.

[0089] The processor is used to execute programs, specifically the relevant steps in the above method embodiments.

[0090] Specifically, the program may include program code, which includes computer operation instructions.

[0091] The processor may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a smart device may be of the same type, such as one or more CPUs; or they may be of different types, such as one or more CPUs and one or more ASICs.

[0092] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0093] The program may include multiple computer instructions, and specifically, the program may use multiple computer instructions to cause the processor to execute the model training method or tone mapping method described in any of the foregoing multiple method embodiments.

[0094] The specific implementation of each step in the program can be found in the corresponding descriptions of the steps, modules, or units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device, equipment, or module can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0095] This invention also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0096] This invention also provides a computer program product, including computer instructions that instruct a computing device to execute the model training method or tone mapping method in the above-described multiple method embodiments.

[0097] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with relevant regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0098] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of the present invention can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0099] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and subsequently stored in a local recording medium, downloaded via a network. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0100] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present invention.

[0101] The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A model training method, comprising: Acquire training data, which includes tonal image samples and multi-scale visual structure data of the tonal image samples. The tonal image samples are LDR image samples, and the multi-scale visual structure data are MSCN data after the tonal image samples have undergone mean subtraction and contrast normalization. A diffusion model is determined, which includes a pre-training module, a pre-configured parsing module, and an encoding module; Using the hue image samples as the main input and output of the diffusion model, and the multi-scale visual structure data as the conditional input of the pre-configured parsing module, the network parameters of the encoding module are iteratively trained at least to obtain a hue mapping model. The iterative training of the network parameters of the encoding module includes: the pre-configured parsing module determining the visual structure features of the initial iteration as the shared visual structure features for each iteration; in each iteration, the pre-configured parsing module performs visual structure feature parsing on the context feature map of that iteration to generate the hue features and visual structure features of that iteration; and the pre-configured parsing module uses the multi-scale visual structure data as a reconstruction condition to fuse the hue features of that iteration with the shared visual structure features to obtain a reconstructed feature map. The pre-training module is used to generate a context feature map based on the random noise addition result of the tone image sample. The pre-configuration parsing module is used to perform visual structural feature parsing on the context feature map and use the multi-scale visual structural data as the reconstruction condition to perform visual structural feature reconstruction on the parsing result to obtain a reconstructed feature map. The encoding module is used to perform latent space encoding on the reconstructed feature map to obtain the tone image sample.

2. The method according to claim 1, wherein, The pre-training module includes a generator network and a control network, wherein at least the network parameters of the encoding module are iteratively trained, including: While iteratively training the network parameters of the encoding module, the control network is fine-tuned by using at least the multi-scale visual structure data as a conditional input to the control network.

3. The method according to claim 2, wherein, The training data also includes blurred brightness data of the tonal image samples, wherein the control network is fine-tuned using at least the multi-scale visual structure data as conditional input, including: The control network is fine-tuned by using the multi-scale visual structure data and the blurred brightness data as the first and second conditional inputs, respectively.

4. The method according to claim 3, wherein, Using the multi-scale visual structure data and the blurred brightness data as the first and second conditional inputs to the control network, respectively, the control network is fine-tuned and trained, including: The visual structural features are fed into the first encoding layer of the control network as a first conditional input, and the blurred brightness data is fed into the second encoding layer of the control network as a second conditional input to fine-tune the control network. In each encoding layer of the control network, the first encoding layer is located on the input side of the second encoding layer.

5. The method according to claim 3, wherein, Each decoding layer of the control network is integrated into the corresponding decoding layer of the generation network.

6. A tone mapping method, comprising: A tone mapping model is obtained, the tone mapping model including a pre-training module, a pre-configured parsing module and an encoding module, the tone mapping model being trained by the method according to any one of claims 1-5; Visual structure processing is performed on the high-tone image to obtain multi-scale visual structure data of the high-tone image; The high-tone image is fed into the main input of the tone mapping model, and the multi-scale visual structure data is fed into the conditional input of the pre-configured parsing module, and the low-tone image is output from the tone mapping model.

7. The method according to claim 6, wherein, The method further includes: The high-key image is subjected to brightness processing to obtain blurred brightness data of the high-key image, wherein, When outputting a low-tone image from the tone mapping model, the multi-scale visual structure data and the blurred brightness data are fed into the first conditional input and the second conditional input of the control network in the pre-trained module.

8. A model training device, comprising: The acquisition unit acquires training data, which includes tone image samples and multi-scale visual structure data of the tone image samples. The tone image samples are LDR image samples, and the multi-scale visual structure data are MSCN data after the tone image samples have undergone mean subtraction and contrast normalization processing. The unit is determined, and the diffusion model is determined, wherein the diffusion model includes a pre-training module, a pre-configuration parsing module, and an encoding module; The training unit uses the hue image samples as the main input and output of the diffusion model, and the multi-scale visual structure data as the conditional input of the pre-configured parsing module. Iterative training is performed on the network parameters of the encoding module at least once to obtain a hue mapping model. The iterative training of the network parameters of the encoding module includes: the pre-configured parsing module determining the visual structure features of the initial iteration as shared visual structure features for each iteration; in each iteration, the pre-configured parsing module performs visual structure feature parsing on the context feature map of that iteration to generate hue features and visual structure features for that iteration; the pre-configured parsing module uses the multi-scale visual structure data as reconstruction conditions to fuse the hue features of that iteration with the shared visual structure features to obtain a reconstructed feature map; the pre-training module generates a context feature map based on the random noise addition result of the hue image samples; the pre-configured parsing module performs visual structure feature parsing on the context feature map and uses the multi-scale visual structure data as reconstruction conditions to reconstruct visual structure features from the parsing result to obtain a reconstructed feature map; and the encoding module performs latent space encoding on the reconstructed feature map to obtain the hue image samples.

9. A tone mapping device, comprising: The model acquisition unit acquires a tone mapping model, which includes a pre-training module, a pre-configuration parsing module, and an encoding module. The tone mapping model is trained by the method according to any one of claims 1-5. The image processing unit performs visual structure processing on the high-tone image to obtain multi-scale visual structure data of the high-tone image. The tone mapping unit feeds at least the high-tone image into the main input of the tone mapping model and feeds the multi-scale visual structure data into the conditional input of the pre-configured parsing module, and outputs a low-tone image from the tone mapping model.

10. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the method according to any one of claims 1-7.

11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1-7.

12. A computer program product comprising computer instructions that, when executed by a processor, implement the method of any one of claims 1-7.

Images