Image processing method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides an image processing method, electronic equipment, a storage medium and a program product. The image processing method comprises the following steps: determining the original pixel brightness of a current image according to image data to be processed; dividing the original pixel brightness into a plurality of data branches; obtaining a histogram corresponding to the original pixel brightness of each data branch; determining a tone mapping curve of each data branch based on a tone mapping model and the histogram; fusing the tone mapping curve of each data branch to obtain a fusion curve; obtaining target pixel brightness corresponding to the original pixel brightness based on the fusion curve; and determining target image data corresponding to the to-be-processed image data based on the original pixel brightness and the target pixel brightness. According to the scheme of the embodiment, automatic tone mapping is realized, and the flexibility and the adaptability of tone mapping are improved.

Description

Technical Field

[0001] This disclosure relates to the fields of image processing and computer vision, and particularly to an image processing method, electronic device, storage medium, and program product. Background Technology

[0002] Among related technologies, tone mapping technology is widely used in various computer vision tasks. One typical application is converting high dynamic range (HDR) images into standard dynamic range (SDR) images. This technology is crucial in many application scenarios, such as mobile phone photography, image signal processing (ISP) in automotive cockpits, video surveillance, and satellite image analysis.

[0003] Traditional tone mapping techniques rely on hand-designed curves, which, while performing well in specific situations, lack flexibility and adaptability. Summary of the Invention

[0004] This disclosure provides an image processing method, an electronic device, a storage medium, and a program product.

[0005] In a first aspect, embodiments of this disclosure provide an image processing method, comprising:

[0006] Determine the original pixel brightness of the current image based on the image data to be processed;

[0007] The original pixel brightness is divided into multiple data branches;

[0008] Obtain the histogram corresponding to the original pixel brightness for each of the various data branches;

[0009] The tone mapping curve corresponding to each data branch is determined based on the preset tone mapping model and the histogram.

[0010] The tone mapping curves corresponding to each data branch are fused to obtain a fused curve;

[0011] Based on the fusion curve, the target pixel brightness corresponding to the original pixel brightness after tone mapping is obtained;

[0012] The target image data corresponding to the image data to be processed, after tone mapping, is determined based on the original pixel brightness and the target pixel brightness.

[0013] Secondly, embodiments of this disclosure also provide an electronic device, including:

[0014] One or more processors;

[0015] A memory having stored one or more programs that, when executed by one or more processors, enable the one or more processors to implement the image processing method.

[0016] One or more input / output (I / O) interfaces are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

[0017] Thirdly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method described above.

[0018] Fourthly, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the image processing method.

[0019] This embodiment of the scheme determines the original pixel brightness of the current image based on the image data to be processed; divides the original pixel brightness into multiple data branches to implement multiple data processing methods to adapt to image data with different characteristics; obtains the histogram corresponding to the original pixel brightness of each data branch; determines the tone mapping curve corresponding to each data branch based on a preset tone mapping model and the histogram, realizing automated tone mapping, and the tone mapping curve obtained by the tone mapping model through deep learning can better capture the local features and global information of the image, and the tone mapping model has high computational efficiency and can be quickly applied to real-time image processing; fuses the tone mapping curves corresponding to each data branch to obtain a fusion curve; obtains the target pixel brightness corresponding to the original pixel brightness after tone mapping based on the fusion curve; and determines the target image data corresponding to the image data to be processed after tone mapping based on the original pixel brightness and the target pixel brightness. This embodiment of the scheme realizes dynamic adjustment of tone mapping strategy according to the specific content and characteristics of the input image, and can handle more complex image processing scenarios, thereby achieving higher adaptability and flexibility. Attached Figure Description

[0020] In the accompanying drawings of the embodiments disclosed herein:

[0021] Figure 1 This is a flowchart of an image processing method provided in an embodiment of the present disclosure;

[0022] Figure 2 This is a schematic diagram of an image processing method provided in an embodiment of the present disclosure;

[0023] Figure 3 Macro-pixel diagrams for four Bayer modes provided for related technologies;

[0024] Figure 4 A schematic diagram illustrating a method for tone mapping of histogram data from the previous frame, provided in an embodiment of this disclosure;

[0025] Figure 5 A flowchart illustrating the method for obtaining a tone mapping model provided in this embodiment of the disclosure;

[0026] Figure 6 A flowchart of a method for acquiring training data provided in an embodiment of this disclosure;

[0027] Figure 7 A flowchart illustrating a method for obtaining multiple linear global tone mapping (GTM) curves and multiple logarithmic GTM curves based on standard sample images, provided in this embodiment of the disclosure.

[0028] Figure 8 A schematic diagram of the network architecture of the micro-learning convolutional neural network provided in the embodiments of this disclosure;

[0029] Figure 9 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0030] To enable those skilled in the art to better understand the technical solutions of this disclosure, the communication-sensing data processing method and computer-readable storage medium provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0031] The present disclosure will be described more fully below with reference to the accompanying drawings; however, the embodiments shown may be embodied in different forms, and the present disclosure should not be construed as limited to the embodiments set forth below. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of the disclosure.

[0032] The accompanying drawings of the embodiments disclosed herein are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the detailed embodiments to explain this disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the description of the detailed embodiments with reference to the accompanying drawings.

[0033] This disclosure may be described with reference to plan and / or cross-sectional views using the ideal schematic diagrams of this disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances.

[0034] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0035] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. The term "and / or" as used in this disclosure includes any and all combinations of one or more of the associated enumerated entries. The singular forms "a" and "the" as used in this disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. The terms "comprising," "made of," etc., as used in this disclosure specify the presence of the stated feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0036] Unless otherwise specified, all terms used in this disclosure (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined in this disclosure.

[0037] Among related technologies, tone mapping technology is widely used in various computer vision tasks. One typical application is converting high dynamic range (HDR) images into standard dynamic range (SDR) images. This technology is crucial in many application scenarios, such as mobile phone photography, image signal processing (ISP) in automotive cockpits, video surveillance, and satellite image analysis.

[0038] Traditional tone mapping techniques rely on hand-designed curves, which, while performing well in specific situations, lack flexibility and adaptability.

[0039] This embodiment of the scheme determines the original pixel brightness of the current image based on the image data to be processed; divides the original pixel brightness into multiple data branches to implement multiple data processing methods to adapt to image data with different characteristics; obtains the histogram corresponding to the original pixel brightness of each data branch; determines the tone mapping curve corresponding to each data branch based on a preset tone mapping model and the histogram, realizing automated tone mapping, and the tone mapping curve obtained by the tone mapping model through deep learning can better capture the local features and global information of the image, and the tone mapping model has high computational efficiency and can be quickly applied to real-time image processing; fuses the tone mapping curves corresponding to each data branch to obtain a fusion curve; obtains the target pixel brightness corresponding to the original pixel brightness after tone mapping based on the fusion curve; and determines the target image data corresponding to the image data to be processed after tone mapping based on the original pixel brightness and the target pixel brightness. This embodiment of the scheme realizes dynamic adjustment of tone mapping strategy according to the specific content and characteristics of the input image, and can handle more complex image processing scenarios, thereby achieving higher adaptability and flexibility.

[0040] The solutions disclosed herein can be applied to any application scenario requiring tone mapping, including but not limited to applications in the ISP (Image Signal Processor) of a terminal, such as mobile phones, computers, tablets, smart wearable devices, in-vehicle devices, smart home devices, and monitoring devices with displays. The solutions disclosed herein can also be extended to other fields requiring tone mapping technology, such as virtual reality (VR) and augmented reality (AR).

[0041] The solutions disclosed herein are particularly applicable to HDR (High Dynamic Range) mode and normal mode. 1. Tone Mapping in HDR Mode: In HDR mode, images typically contain a very wide brightness range, which may cause some details to be lost in highlight or shadow areas. The solutions disclosed herein intelligently analyze image content, identify highlight and shadow areas, and then appropriately adjust brightness and contrast to preserve details in these areas. By using a tone mapping model, different areas in the image can be identified more accurately, and different mapping strategies can be applied, thereby achieving a more balanced and natural visual experience. 2. Image Enhancement in Normal Mode: In normal mode, although the dynamic range of the image may not be as wide as in HDR mode, contrast enhancement remains key to improving image quality. The solutions disclosed herein enhance image contrast by analyzing the statistical characteristics of the image, making the image appear clearer and more vivid. Contrast enhancement not only improves the visual appeal of the image but also helps improve image consistency across different display devices.

[0042] The embodiments of this disclosure will be described in detail below.

[0043] This disclosure provides an image processing method, such as... Figure 1 , Figure 2 As shown, steps S11-S17 are included:

[0044] S11. Determine the original pixel brightness of the target pixel based on the image data to be processed.

[0045] In this embodiment of the disclosure, the estimation of the tone mapping curve is performed in the luminance domain. Therefore, it is first necessary to estimate the original pixel luminance L of the target pixel based on the input image data to be processed (e.g., red, green, blue RGB data), which can be achieved through a preset luminance estimation module.

[0046] In this embodiment of the disclosure, if the luminance estimation module is placed in the YUV (luminance, chrominance, density) domain, the Y value can be directly referenced, that is, L = Y.

[0047] In this embodiment of the disclosure, if the brightness estimation module is placed in the RGB domain and the input is sRGB (standard red, green, blue) data, the original pixel brightness L can be calculated by the following first calculation formula: L = 2.99R + 0.587G + 0.114B.

[0048] In this embodiment of the disclosure, if the brightness estimation module is placed on the Bayer domain and the input is RAW (raw material) data, the original pixel brightness L is calculated according to the following second calculation formula: Among them, R and G r G b The four variables B, A, and B correspond to the values ​​of the four channels in a 2x2 macropixel in Bayer mode, such as... Figure 3 As shown, the second calculation formula is actually an approximation of the first calculation formula.

[0049] S12. Divide the original pixel brightness into multiple data branches.

[0050] In this embodiment of the disclosure, the various data branches can be defined according to requirements, and the types of data branches are not limited in detail here.

[0051] In this embodiment of the disclosure, various data branches may include, but are not limited to, linear data branches and logarithmic data branches.

[0052] In this embodiment of the disclosure, to accommodate different types of data analysis needs, the raw pixel brightness data stream can be divided into two data branches: a linear branch (i.e., linear data branch) and a logarithmic branch (i.e., logarithmic data branch). These two data branches represent two different processing paths, each corresponding to different data characteristics and analysis objectives.

[0053] In this embodiment of the disclosure, the linear data branches can be directly subjected to histogram statistics. This processing method is suitable for situations where the data changes relatively evenly, and the distribution of the data can be seen intuitively.

[0054] In this embodiment of the disclosure, the logarithmic data branch needs to first convert the original pixel brightness L to the logarithmic domain. This conversion can be performed based on the following conversion formula, and then histogram statistics are performed. The logarithmic data branch is closer to the natural response of the human eye to light.

[0055] In this embodiment of the disclosure, the conversion calculation formula is as follows:

[0056]

[0057] Among them, L lo The bit_width represents the original pixel brightness corresponding to the logarithmic data branch; bit_width is the sampling depth of the input RGB data or Bayer data.

[0058] In this embodiment of the disclosure, the processing of L+1 is to avoid the independent variable taking the value of 0, while the fraction... It is used to adjust the range of values, ensuring calculation accuracy while keeping the range of values ​​consistent with the linear data branches, which is beneficial for the next step of histogram statistics.

[0059] In this embodiment, two processing modes, linear data branching and logarithmic data branching, are combined to adapt to image data with different characteristics. The linear mode is suitable for situations where data changes relatively uniformly, maintaining the relative proportions of the original data. The logarithmic mode more closely resembles the natural response of the human eye to light, easily producing images that are more natural and user-friendly for human observers.

[0060] S13. Obtain the histogram corresponding to the original pixel brightness of each data branch in the multiple data branches.

[0061] In this embodiment of the disclosure, each histogram is used to characterize the pixel brightness of the corresponding data branch into multiple regions divided according to the size of the brightness domain.

[0062] In this embodiment, the number of bins (bars) is first determined to be K (K is a positive integer). Then, the brightness domain of the original pixel brightness is evenly divided into K regions from the minimum to the maximum value, and the following formula is used for statistical analysis:

[0063]

[0064] Where k∈[0,K), the formula above defines K intervals, which are uniformly distributed and divided into K equal parts between the minimum value (minimum original pixel brightness) and the maximum value (maximum original pixel brightness). The value of each interval includes the left endpoint but not the right endpoint. L log_max L represents the maximum value of the original pixel brightness corresponding to the logarithmic data branch. linear L represents the original pixel brightness corresponding to the linear data branch. linear_max The maximum value of the original pixel brightness corresponding to the linear data branch; bin k This is the k-th bin.

[0065] S14. Determine the tone mapping curve corresponding to each data branch based on the preset tone mapping model and histogram.

[0066] In this embodiment of the disclosure, determining the tone mapping curve corresponding to each data branch can be implemented in a preset mapping module.

[0067] In this embodiment of the disclosure, determining the tone mapping curve corresponding to each data branch based on a preset tone mapping model and histogram may include:

[0068] Obtain the histogram of the previous frame, input the tone mapping model, and obtain the tone mapping curve corresponding to the histogram data of the previous frame.

[0069] In this embodiment, the mapping module can be integrated into the ISP pipeline. Due to constraints such as hardware implementation cost and real-time performance, the pipeline may not be able to buffer an entire frame of data during processing (each frame contains histogram data). This limitation means that it is not possible to directly analyze the data of the current frame in real time. To solve this problem, a clever method can be adopted: the generation of the tone mapping curve is delayed by one frame accordingly. That is, the data processing of the current frame is actually based on the histogram data of the previous frame in memory, such as... Figure 4 As shown, this method enables the optimization of image processing using data from each frame while maintaining the pipeline's high efficiency. This method ensures effective image data processing even with limited hardware resources. Through this approach, the ISP pipeline can achieve accurate and efficient image data processing while maintaining cost-effectiveness and meeting real-time requirements.

[0070] In the embodiments disclosed herein, such as Figure 5 As shown, the method for obtaining the tone mapping model may include steps S21-S24:

[0071] S21. Obtain training data.

[0072] In this embodiment of the disclosure, the training data may include multiple data pairs, which may include multiple tone mapping curves and corresponding histograms. The types of tone mapping curves are the same as the types of data branches. For example, the data pairs may include, but are not limited to: data pairs consisting of linear tone mapping curves and corresponding linear histograms, and data pairs consisting of logarithmic tone mapping curves and corresponding logarithmic histograms.

[0073] In the embodiments disclosed herein, such as Figure 6 As shown, obtaining training data may include steps S31-S36:

[0074] S31. Collect sample images.

[0075] In this embodiment of the disclosure, a target device, such as a specific model of mobile phone, can be used to capture RAW format images as the sample images.

[0076] S32. Perform preprocessing on the sample image and map it to the standard red-green-blue sRGB space to obtain a standard sample image.

[0077] In this embodiment of the disclosure, the sample image can be converted to sRGB space (before tone mapping) after a series of preprocessing steps by the ISP to obtain a standard sample image and export it.

[0078] S33. Obtain multiple global tone mapping (GTM) curves based on standard sample images; wherein, the multiple GTM curves serve as the multiple tone mapping curves.

[0079] In this embodiment of the disclosure, obtaining multiple global tone mapping (GTM) curves based on standard sample images may include:

[0080] Multiple linear global tone mapping (GTM) curves and multiple logarithmic GTM curves were obtained based on standard sample images.

[0081] In the embodiments disclosed herein, such as Figure 7 As shown, obtaining multiple linear global tone mapping (GTM) curves and multiple logarithmic GTM curves based on standard sample images may include steps S41-S44:

[0082] S41. Obtain the GTM curve of the standard sample image.

[0083] S42. Obtain the target GTM curve after manual adjustment.

[0084] In this embodiment of the disclosure, the GTM curve of the obtained sRGB image (i.e., standard sample image) can be manually adjusted until a satisfactory result (i.e., target GTM curve) is obtained, and these results are saved.

[0085] S43. Based on each target GTM curve, randomly generate multiple corresponding similar GTM curves; the similar GTM curves are GTM curves whose similarity to the target GTM curve is less than a preset similarity threshold.

[0086] In this embodiment of the disclosure, for each well-tuned result, S (S is a positive integer, such as S=5) similar GTM curves can be randomly generated.

[0087] S44. Decompose multiple similar GTM curves into multiple linear GTM curves and multiple logarithmic GTM curves.

[0088] In this embodiment of the disclosure, the fusion weights can be randomly decomposed into S linear GTM curves and S logarithmic GTM curves.

[0089] S34. Obtain the histograms corresponding to various GTM curves respectively.

[0090] In this embodiment of the disclosure, obtaining histograms corresponding to various GTM curves may include:

[0091] Obtain linear histograms corresponding to multiple linear GTM curves, and obtain logarithmic histograms corresponding to multiple logarithmic GTM curves.

[0092] In this embodiment of the disclosure, the inverse mapping curves of the S+S GTM curves can be obtained respectively, and then these inverse mapping curves can be used to map each debugged target GTM curve into M+M random images, and the linear histogram and logarithmic histogram corresponding to the S+S random images can be calculated.

[0093] In the embodiments of this disclosure, the methods for obtaining linear histograms and logarithmic histograms will be described in detail below.

[0094] In this embodiment of the disclosure, obtaining a linear histogram corresponding to multiple linear GTM curves may include:

[0095] Obtain the first inverse mapping curves corresponding to multiple linear GTM curves;

[0096] Based on multiple first inverse mapping curves, the image corresponding to the target GTM curve is mapped to multiple first random images;

[0097] Histogram statistics are performed on the first random image to obtain a linear histogram.

[0098] In this embodiment of the disclosure, obtaining logarithmic histograms corresponding to multiple logarithmic GTM curves includes:

[0099] Obtain the second inverse mapping curves corresponding to multiple logarithmic GTM curves;

[0100] Based on multiple second inverse mapping curves, the image corresponding to the target GTM curve is mapped to multiple second random images;

[0101] Histogram statistics are performed on the second random image to obtain a logarithmic histogram.

[0102] S35. Combine various GTM curves with their corresponding histograms to form data pairs.

[0103] In this embodiment of the disclosure, the data pairs may include, but are not limited to, linear data pairs and logarithmic data pairs; forming data pairs by combining multiple GTM curves with their corresponding histograms may include:

[0104] Linear data pairs are formed by combining the linear GTM curve with its corresponding linear histogram, and logarithmic data pairs are formed by combining the logarithmic GTM curve with its corresponding logarithmic histogram.

[0105] In this embodiment of the disclosure, a linear tone mapping curve and a corresponding linear histogram are used to form a linear data pair, and a logarithmic tone mapping curve and a corresponding logarithmic histogram are used to form a logarithmic data pair.

[0106] S36. Use multiple data pairs as training data.

[0107] In this embodiment of the disclosure, multiple data pairs consisting of histograms and GTM curves can be used as training data for training a micro-learning convolutional neural network.

[0108] S22. Train the preset micro-learning convolutional neural network based on each data pair in each data pair in the training data, and obtain the corresponding loss value respectively.

[0109] In this embodiment of the disclosure, the micro-learning convolutional neural network (i.e., a miniature CNN network) uses a network architecture that mainly includes two convolutional layers and one fully connected layer. For example... Figure 8As shown, the detailed network architecture of a micro-learning convolutional neural network may include, but is not limited to: an input layer, a first convolutional layer (Convolution1), a second convolutional layer (Convolution2), an activation layer (ReLU), a pooling layer, a fully connected layer, and an output layer (Sigmoid). Using a convolutional neural network (CNN) to generate tone mapping curves can be viewed as a regression problem, that is, predicting the quantitative relationship between the input data (in this embodiment, the histogram statistics of the image) and the output data (tone mapping curve).

[0110] In this embodiment, the input layer of the micro-learning convolutional neural network receives histogram statistics, the hidden layer is responsible for extracting the relationship between features and learning data, and the output layer generates the final tone mapping curve. Through backpropagation, the network learns to adjust the parameters of the micro-learning convolutional neural network to minimize the difference between the predicted output and the actual output. Compared with traditional tone mapping methods, using a CNN network architecture offers greater flexibility and adaptability, enabling the generation of more accurate and personalized tone mapping curves based on different image content and display conditions.

[0111] In this embodiment, by introducing a micro-learning mechanism, a more complex tone mapping curve obtained through deep learning is used instead of the manually crafted tone mapping curve in traditional techniques. This significantly improves the lack of flexibility and adaptability of existing tone mapping technologies. Compared to traditional manually implemented tone mapping curves, tone mapping curves obtained through deep learning can better capture local features and global information of an image. Deep learning models can automatically learn how to adjust the brightness, contrast, and color of an image under different lighting conditions to achieve a more natural effect that better matches human visual perception. Furthermore, they can dynamically adjust tone mapping parameters based on the specific content and characteristics of the input image, thereby achieving higher adaptability and handling more complex scenes, such as high-contrast, multi-light source, or non-uniform lighting conditions.

[0112] In this embodiment of the disclosure, the data pairs may include, but are not limited to, linear data pairs and logarithmic data pairs;

[0113] The predefined micro-learning convolutional neural network is trained on each data pair in the training data, and the corresponding loss value is obtained respectively. This can include:

[0114] The micro-learning convolutional neural network is trained for each linear data pair to obtain the corresponding linear loss value; and,

[0115] The micro-learning convolutional neural network is trained for each log data pair to obtain the corresponding log loss value.

[0116] In this embodiment of the disclosure, during the generation of tone mapping curves, a loss function is used to quantify the difference between the tone mapping curve predicted by the micro-learning convolutional neural network and the ideal tone mapping curve. By optimizing the loss function, the micro-learning convolutional neural network can be guided to learn how to generate tone mapping curves more accurately.

[0117] In this embodiment of the disclosure, the generated linear tone mapping curve is assumed to be... linear Logarithmic tone mapping curve log Each curve is represented by M+1 nodes, which define the key control points of the curve and thus determine the overall shape and characteristics of the tone mapping curve.

[0118] In this embodiment of the disclosure, for each type of tone mapping curve, the N data pairs can be input into a micro-learning convolutional neural network for training, and corresponding loss values ​​can be obtained. Furthermore, after each data pair is input into the micro-learning convolutional neural network, a loss value can be obtained for each of the M+1 nodes. That is, each data pair can correspond to M+1 loss values, and the N data pairs correspond to a total of N(M+1) loss values.

[0119] In this embodiment of the disclosure, for example, N linear data pairs consisting of linear tone mapping curves and corresponding linear histograms can be input into a micro-learning convolutional neural network for training, and corresponding linear loss values ​​can be obtained. Each linear data pair, after being input into the micro-learning convolutional neural network, can also obtain a linear loss value for each of the M+1 nodes. That is, each linear data pair can correspond to M+1 linear loss values, and N linear data pairs can correspond to a total of N(M+1) linear loss values.

[0120] In this embodiment of the disclosure, for example, N logarithmic data pairs consisting of logarithmic tone mapping curves and corresponding logarithmic histograms can be input into a micro-learning convolutional neural network for training, and corresponding logarithmic loss values ​​can be obtained. Specifically, after each logarithmic data pair is input into the micro-learning convolutional neural network, a logarithmic loss value can be obtained for each of the M+1 nodes. That is, each logarithmic data pair can correspond to M+1 logarithmic loss values, and N logarithmic data pairs can correspond to a total of N(M+1) logarithmic loss values.

[0121] S23. Perform a weighted calculation on all the loss values ​​corresponding to each data pair to obtain the total loss value corresponding to each data pair.

[0122] In this embodiment of the disclosure, the total loss value corresponding to each data pair may include a linear total loss value and a logarithmic total loss value.

[0123] In this embodiment of the disclosure, the N(M+1) linear loss values ​​corresponding to N linear data pairs can be comprehensively calculated according to the following first loss value calculation formula to obtain the total linear loss value Loss. linear :

[0124]

[0125] Where m∈[1,M], m refers to the m-th node, n∈[1,N], n refers to the n-th linear data pair, and weight linear [m] is the linear weighting coefficient corresponding to the m-th node, curve linear [m] is the predicted linear tone mapping curve data corresponding to the m-th node, curve linear_gt [m] represents the ideal linear tone mapping curve data corresponding to the m-th node; weight linear [m]×(curve linear [m]-curve linear_gt [m]) is the linear loss value.

[0126] In this embodiment of the disclosure, the N(M+1) logarithmic loss values ​​corresponding to N logarithmic data pairs can be comprehensively calculated according to the second loss value calculation formula described below to obtain the total logarithmic loss value Loss. log :

[0127]

[0128] Where m∈[1,M], m refers to the m-th node, n∈[1,N], n refers to the n-th logarithmic data pair, and weight log [m] is the log-weighted coefficient corresponding to the m-th node, curve log [m] represents the predicted logarithmic tone mapping curve data corresponding to the m-th node, curve log_gt [m] is the ideal logarithmic tone mapping curve data corresponding to the m-th node; weight log [m]×(curve log [m]-curve log_gt [m]) is the logarithmic loss value.

[0129] In this embodiment of the disclosure, the weighting coefficient (or weight value, e.g., weight) linear [m] and weight log [m]) allows different samples to be assigned different levels of importance, which is very useful when dealing with samples of different importance. The specific value can be determined by experiments, prior knowledge, the importance of the data, or specific business needs.

[0130] In this embodiment, the loss function calculates multiple loss values ​​by weighting them according to preset weighting coefficients. By introducing weighting coefficients, different parts of the loss function can be weighted, thereby optimizing the micro-learning convolutional neural network's learning of specific features. The weighted loss function can guide the micro-learning convolutional neural network to pay more attention to errors with higher weights, improving the performance of the final tone mapping model on specific tasks.

[0131] S24. If the total loss value for each data pair meets the preset requirements, save the network parameters of the micro-learning convolutional neural network to obtain the tone mapping model.

[0132] In this embodiment of the disclosure, the preset requirement can be defined by the user, and no specific details of the preset requirement are limited herein.

[0133] In this embodiment of the disclosure, the preset requirement may include a first preset requirement and a second preset requirement, which may be the same or different. In the linear total loss value Loss linear The first preset requirement is met and the total logarithmic loss value is Loss log If the second preset requirement is met, the network parameters of the micro-learning convolutional neural network at this time can be saved. The micro-learning convolutional neural network based on these network parameters is the final tone mapping model.

[0134] In this embodiment, after obtaining the tone mapping model, a tone mapping curve can be directly generated based on the histogram. The tone mapping model refers to a small convolutional neural network capable of learning the mapping relationship from an image's histogram to a tone mapping curve. This model learns the relationship between different histogram features and the ideal tone mapping curve through training, thereby achieving automated tone mapping. The advantage of the tone mapping model lies in its high computational efficiency, allowing for rapid application in real-time image processing.

[0135] S15. Merge the tone mapping curves corresponding to each data branch to obtain the merged curve.

[0136] In this embodiment of the disclosure, a linear tone mapping curve can be obtained through the aforementioned steps. linear And logarithmic tone mapping curve log To obtain the final tone mapping curve, the two tone mapping curves need to be blended in a certain proportion. The blending calculation formula is as follows:

[0137] curve tm = weight × curve linear +(1-weight)×curve log ;

[0138] Among them, curve tm The curve is called the fusion curve, and weight is the fusion weight, which is a parameter that needs to be calibrated in advance.

[0139] S16. Obtain the target pixel brightness after tone mapping, corresponding to the original pixel brightness, based on the fusion curve.

[0140] In this embodiment of the disclosure, curve is obtained tm After the curve is drawn, the target pixel brightness L after tone mapping can be obtained from the original pixel brightness L. tm If the original pixel brightness L is exactly equal to the curve tm The value of the independent variable at a certain node of the curve, then L tm The value can be obtained directly from a table. If the original pixel brightness L is not equal to the curve... tm The value of the independent variable at a certain node of the curve can be used to obtain the target pixel brightness L after tone mapping through linear interpolation. tm .

[0141] S17. Determine the target image data after tone mapping corresponding to the image data to be processed based on the original pixel brightness and the target pixel brightness.

[0142] In this embodiment of the disclosure, determining the target image data after tone mapping corresponding to the image data to be processed based on the original pixel brightness and the target pixel brightness includes:

[0143] The target image data is determined based on a preset image mapping formula and the image data to be processed; wherein, the image mapping formula includes the original pixel brightness and the target pixel brightness.

[0144] In this embodiment of the disclosure, the image mapping calculation formula may include, but is not limited to:

[0145]

[0146] Among them, RGB output For target image data, RGB input The image data to be processed; L tm L represents the target pixel brightness, and L represents the original pixel brightness.

[0147] This disclosure embodiment includes at least the following advantages:

[0148] 1. Adaptability: The solution of this disclosure can automatically detect image content and environmental conditions, and adjust tone mapping parameters in real time to adapt to different lighting and scenes. Through deep learning technology, the solution of this disclosure can learn the optimal mapping strategy under different conditions, thereby achieving more refined and accurate dynamic range compression. In addition, through two processing modes, linear data branching and logarithmic data branching, it can adapt to image data with different characteristics.

[0149] 2. Flexibility: The solutions of this disclosure support custom settings, allowing users to fine-tune tone mapping parameters to meet personalized needs.

[0150] 3. Hardware Overhead: The embodiments of this disclosure fully consider the limitations of hardware resources, optimize computational complexity and memory usage, and ensure efficient operation even on low-power and low-cost hardware. By simplifying the model structure and adopting efficient algorithm implementation, the demand for computation and memory is reduced.

[0151] 4. Compatibility with ISP: The solution of this disclosure is compatible with existing image signal processor (ISP) architectures and can be seamlessly integrated into various devices without requiring large-scale modifications to existing hardware.

[0152] This disclosure also provides an electronic device 100, such as... Figure 9 As shown, it includes:

[0153] One or more processors 101;

[0154] The memory 102 stores one or more programs, which, when executed by one or more processors 101, enable the one or more processors to implement the image processing method described above.

[0155] One or more input / output I / O interfaces 103 are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

[0156] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method described above.

[0157] This disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the image processing method described above.

[0158] Those skilled in the art will understand that all or some of the functional modules / units disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0159] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be executed by several physical components working together.

[0160] Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only optical disc (CD-ROM), digital versatile disc (DVD) or other optical disc storage; magnetic cartridges, magnetic tapes, disk storage or other magnetic storage; and any other media that can be used to store desired information and can be accessed by a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0161] This disclosure has disclosed exemplary embodiments, and although specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. An image processing method, comprising: Determine the original pixel brightness of the target pixel based on the image data to be processed; The original pixel brightness is divided into multiple data branches; Obtain the histogram corresponding to the original pixel brightness for each of the various data branches; The tone mapping curve corresponding to each data branch is determined based on the preset tone mapping model and the histogram. The tone mapping curves corresponding to each data branch are fused to obtain a fused curve; Based on the fusion curve, the target pixel brightness corresponding to the original pixel brightness after tone mapping is obtained; The target image data corresponding to the image data to be processed, after tone mapping, is determined based on the original pixel brightness and the target pixel brightness.

2. The image processing method according to claim 1, wherein, The determination of the tone mapping curve corresponding to each data branch based on the preset tone mapping model and the histogram includes: Obtain the histogram of the previous frame, input it into the tone mapping model, and obtain the tone mapping curve corresponding to the histogram data of the previous frame.

3. The image processing method according to claim 1, wherein, The various data branches include linear data branches and logarithmic data branches.

4. The image processing method according to claim 1, wherein, The method for obtaining the tone mapping model includes: Obtain training data; The preset micro-learning convolutional neural network is trained based on each data pair in each data pair in the training data, and the corresponding loss value is obtained respectively. For each data pair obtained, a weighted average of all the loss values ​​corresponding to each data pair is calculated to obtain the total loss value corresponding to each data pair. If the total loss value for each data pair meets the preset requirements, the network parameters of the micro-learning convolutional neural network are saved to obtain the tone mapping model.

5. The image processing method according to claim 4, wherein, The acquisition of training data includes: Acquire sample images; The sample image is preprocessed and mapped to the standard red-green-blue sRGB space to obtain a standard sample image; Multiple global tone mapping (GTM) curves are obtained based on the standard sample images; wherein, the multiple GTM curves serve as the multiple tone mapping curves. Obtain the histograms corresponding to the various GTM curves respectively; The various GTM curves are paired with their corresponding histograms to form data pairs. Multiple data pairs are used as the training data.

6. The image processing method according to claim 5, wherein, The process of obtaining multiple global tone mapping (GTM) curves based on the standard sample images includes: Multiple linear global tone mapping (GTM) curves and multiple logarithmic GTM curves are obtained based on the standard sample images.

7. The image processing method according to claim 6, wherein, The process of obtaining multiple linear global tone mapping (GTM) curves and multiple logarithmic GTM curves based on the standard sample images includes: Obtain the GTM curve of the standard sample image; The target GTM curve is obtained by manually adjusting the GTM curve. Based on each target GTM curve, multiple corresponding similar GTM curves are randomly generated; the similar GTM curves are GTM curves whose similarity to the target GTM curve is less than a preset similarity threshold. The multiple similar GTM curves are decomposed into multiple linear GTM curves and multiple logarithmic GTM curves.

8. The image processing method according to claim 6, wherein, The step of obtaining the histograms corresponding to the various GTM curves includes: Obtain linear histograms corresponding to multiple linear GTM curves, and obtain logarithmic histograms corresponding to multiple logarithmic GTM curves.

9. The image processing method according to claim 8, wherein, Obtaining the linear histograms corresponding to multiple linear GTM curves includes: Obtain the first inverse mapping curves corresponding to the multiple linear GTM curves; Based on multiple first inverse mapping curves, the image corresponding to the target GTM curve is mapped into multiple first random images; Histogram statistics are performed on the first random image to obtain the linear histogram.

10. The image processing method according to claim 8, wherein, Obtaining the logarithmic histograms corresponding to multiple logarithmic GTM curves includes: Obtain the second inverse mapping curves corresponding to the multiple logarithmic GTM curves; Based on multiple second inverse mapping curves, the image corresponding to the target GTM curve is mapped into multiple second random images; Histogram statistics are performed on the second random image to obtain the logarithmic histogram.

11. The image processing method according to claim 8, wherein, The data pairs include linear data pairs and logarithmic data pairs; the step of forming data pairs by combining the various GTM curves with their corresponding histograms includes: The linear GTM curve and its corresponding linear histogram are combined to form the linear data pair, and the logarithmic GTM curve and its corresponding logarithmic histogram are combined to form the logarithmic data pair.

12. The image processing method according to claim 4, wherein, The data pairs include linear data pairs and logarithmic data pairs; The process of training a preset micro-learning convolutional neural network based on each data pair in the training data and obtaining a corresponding loss value includes: The micro-learning convolutional neural network is trained on each of the linear data pairs to obtain a corresponding linear loss value; and... The micro-learning convolutional neural network is trained according to each of the logarithmic data pairs to obtain the corresponding logarithmic loss value.

13. The image processing method according to claim 1, wherein, The step of determining the target image data corresponding to the image data to be processed, after tone mapping, based on the original pixel brightness and the target pixel brightness includes: The target image data is determined based on a preset image mapping formula and the image data to be processed; wherein, the image mapping formula includes the original pixel brightness and the target pixel brightness.

14. The image processing method according to claim 13, wherein, The image mapping calculation formula includes: Among them, RGB output The target image data, RGB input The image data to be processed; L tm L represents the target pixel brightness, and L represents the original pixel brightness.

15. An electronic device comprising: One or more processors; A memory having stored one or more programs thereon, which, when executed by the one or more processors, cause the one or more processors to implement the image processing method according to any one of claims 1-14; One or more input / output (I / O) interfaces are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

16. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method according to any one of claims 1-14.

17. A computer program product comprising a computer program that, when executed by a processor, implements the image processing method according to any one of claims 1-14.

Images