Hdri image, dataset generation, and model training method, image sensor

Abstract

The application is suitable for the technical field of image acquisition, and provides a generation method of an HDR image, a data set generation method, a model training method and an image sensor. The method comprises the following steps: acquiring a plurality of frames of to-be-processed LDR images in a Raw domain; inputting the plurality of frames of to-be-processed LDR images into an image processing model to obtain an initial HDR image in the Raw domain which is subjected to tone mapping and output by the image processing model. The embodiment of the application can output the result after tone mapping based on the Raw domain LDR image, can be directly integrated into an existing camera ISP process, and higher-efficiency HDR processing can be realized.

Description

Technical Field

[0001] This application belongs to the field of image technology, and in particular relates to a method for generating HDR images, generating datasets and training models, as well as an image sensor. Background Technology

[0002] With the continuous advancement of display and imaging technologies, High Dynamic Range (HDR) imaging has become a significant innovation in image and video processing. HDR technology enhances image brightness, contrast, and color accuracy, preserving rich details even in scenes with strong contrast, and is widely used in television, film, photography, gaming, and virtual reality. Deep learning-based HDR technology can not only automatically adjust exposure and contrast according to the scene but also effectively reduce ghosting, artifacts, and oversaturation issues present in traditional HDR techniques. This makes Artificial Intelligence (AI) HDR technology a key solution for overcoming current bottlenecks in dynamic scene processing, improving image quality, and optimizing processing speed.

[0003] Most current HDR technologies perform image processing in the RGB domain, which is not convenient for integration into the image sensor's image signal processing (ISP). Summary of the Invention

[0004] This application provides a method for generating HDR images, generating datasets, and training models, as well as an image sensor. It can output the results after tone mapping based on Raw domain LDR images and can be directly integrated into existing camera ISP processes to achieve more efficient HDR processing.

[0005] The first aspect of this application provides a method for generating HDR images, comprising: acquiring multiple frames of LDR images to be processed in the Raw domain; inputting the multiple frames of LDR images to be processed into an image processing model to obtain an initial HDR image in the Raw domain after tone mapping output by the image processing model.

[0006] A second aspect of this application provides a method for generating a training dataset, the training dataset being used to train the image processing model described in the first aspect; the method for generating the training dataset includes:

[0007] Obtain multi-frame sample LDR images of the target object in the Raw domain; obtain sample HDR images of the target object in the RGB domain; perform inverse demosaic processing on the sample HDR images to obtain the target image in the Raw domain; generate a training dataset based on the multi-frame sample LDR images and the target image.

[0008] A third aspect of this application provides a model training method, comprising: acquiring a training dataset, wherein the training dataset is obtained according to the training dataset generation method described in the second aspect; inputting the multi-frame sample LDR images in the training dataset into a model to be trained, and training the model with the goal of minimizing the loss value of the model to be trained, thereby obtaining an image processing model; wherein the loss value and the output result of the model to be trained are related to the difference between the target image and the target image.

[0009] The fourth aspect of this application provides an HDR image generation apparatus, comprising: an acquisition unit for acquiring multiple frames of LDR images to be processed in the Raw domain; and an image generation unit for inputting the multiple frames of LDR images to be processed into an image processing model to obtain an initial HDR image in the Raw domain after tone mapping output by the image processing model.

[0010] A training dataset generation apparatus provided in the fifth aspect of this application includes: a first acquisition unit for acquiring multi-frame sample LDR images of a target object in the Raw domain; a second acquisition unit for acquiring sample HDR images of the target object in the RGB domain; a target image acquisition unit for performing inverse demosaic processing on the sample HDR images to obtain a target image in the Raw domain; and a training dataset generation unit for generating a training dataset based on the multi-frame sample LDR images and the target image.

[0011] A model training apparatus provided in the sixth aspect of this application includes: a training set acquisition unit for acquiring a training dataset; and a training unit for inputting the multi-frame sample LDR images from the training dataset into a model to be trained, and training the model with the goal of minimizing the loss value of the model to be trained, thereby obtaining an image processing model; wherein the loss value and the output result of the model to be trained are related to the difference between the target image and the target image.

[0012] A seventh aspect of this application provides an image sensor, comprising: a data acquisition unit for acquiring multiple frames of LDR images to be processed in the Raw domain; and an image processing unit connected to the data acquisition unit for inputting the multiple frames of LDR images to be processed into an image processing model to obtain an initial HDR image in the Raw domain after tone mapping output by the image processing model.

[0013] An eighth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described training dataset generation method or model training method.

[0014] A ninth aspect of this application provides a computer program product that, when run on an image sensor, causes the image sensor to execute the above-described HDR image generation method.

[0015] The tenth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the above-described training dataset generation method or model training method.

[0016] In the embodiments of this application, by acquiring multiple frames of LDR images to be processed in the Raw domain, inputting these multiple frames of LDR images into an image processing model, and obtaining an initial HDR image in the Raw domain after tone mapping output by the image processing model, the tone-mapped result can be output based on the Raw domain LDR image. This can be directly integrated into the existing camera ISP process to achieve more efficient HDR processing. Furthermore, the image processing model directly outputs the tone-mapped initial HDR image, which is convenient for use on devices with limited processing resources and difficulty in performing tone mapping in the RGB domain, thus helping to improve the applicability of HDR processing. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating the implementation flow of an HDR image generation method provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the conventional ISP process for cameras in existing technology;

[0020] Figure 3 This is a schematic diagram of the ISP process of the camera provided in the embodiments of this application;

[0021] Figure 4 This is a schematic diagram illustrating the specific implementation process of the image processing model provided in this application for processing multiple frames of LDR images;

[0022] Figure 5 This is a schematic diagram of the structure of the image processing model provided in the embodiments of this application;

[0023] Figure 6 This is a schematic diagram of the structure of the attention module provided in an embodiment of this application;

[0024] Figure 7This is a schematic diagram illustrating the implementation flow of the training dataset generation method provided in the embodiments of this application;

[0025] Figure 8 This is a schematic diagram of the installation of the digital camera and sensor provided in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram illustrating the specific implementation process of obtaining the target image provided in the embodiments of this application;

[0027] Figure 10 This is a flowchart illustrating an example of obtaining training data from a training dataset, as provided in an embodiment of this application.

[0028] Figure 11 This is a schematic diagram illustrating the specific implementation process of obtaining sample HDR images provided in the embodiments of this application;

[0029] Figure 12 This is a schematic diagram illustrating the implementation process of a model training method provided in an embodiment of this application;

[0030] Figure 13 This is a schematic diagram of the structure of the HDR image generation apparatus provided in the embodiments of this application;

[0031] Figure 14 This is a schematic diagram of the structure of the training dataset generation device provided in the embodiments of this application;

[0032] Figure 15 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0033] Figure 16 This is a schematic diagram of the structure of the image sensor provided in the embodiments of this application;

[0034] Figure 17 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are protected by this application. It should be understood that, when used in this specification and appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0036] In the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0038] With the continuous advancement of display and imaging technologies, HDR has become a significant innovation in the field of image and video processing. HDR technology enhances the brightness, contrast, and color accuracy of images, preserving rich details even in scenes with strong contrasts, and is widely used in television, film, photography, gaming, and virtual reality. Deep learning-based HDR technology can not only automatically adjust exposure and contrast according to the scene, but also effectively reduce ghosting, artifacts, and oversaturation problems found in traditional HDR technologies. This makes AI HDR technology a key solution for addressing current bottlenecks in dynamic scene processing, improving image quality, and optimizing processing speed.

[0039] Multi-frame HDR fusion is one of the most common HDR techniques. It fuses images exposed in succession to achieve an HDR effect. However, due to the time intervals between frames of varying exposures, the fused image can suffer from misalignment and artifacts. Furthermore, to display HDR images on low dynamic range devices, tone mapping (TMO) is required. This compresses highlight and shadow areas of the HDR image towards medium brightness. However, mobile devices and embedded systems have limited computing resources, making it difficult to integrate complex tone mapping, thus limiting the HDR display effect. Moreover, most publicly available datasets are in RGB format, requiring models trained on such data to perform image processing in the RGB domain, making them unsuitable for direct integration into the image sensor's ISP process.

[0040] In view of the above findings, this application proposes a method for generating HDR images, generating datasets, and training models, as well as an image sensor. It can output the results after tone mapping based on Raw domain LDR images and can be directly integrated into existing ISP processes to achieve more efficient and reliable HDR processing.

[0041] To illustrate the technical solution of this application, specific embodiments are described below.

[0042] Figure 1 The illustration shows a schematic diagram of the implementation process of an HDR image generation method provided in an embodiment of this application. This method can be applied to image sensors.

[0043] It is understood that the image sensor can be a standalone optical instrument or integrated into a mobile device, wearable device, image sensor or other smart device, and this application does not impose any restrictions on this.

[0044] Specifically, the above-mentioned method for generating HDR images may include the following steps S101 to S102.

[0045] Step S101: Obtain multiple frames of LDR images to be processed in the Raw domain.

[0046] In this context, the image in the Raw domain refers to the raw data output from the sensor. For example, it can be a digital signal obtained by quantizing the analog signal output from the pixel array by the analog-to-digital converter circuit of an image sensor (such as a CIS). This digital signal corresponds one-to-one with the pixel unit in the pixel array that acquires photoelectric signals. LDR image refers to Low-Dynamic Range (LDR) image. In the embodiments of this application, multiple frames of LDR images in the Raw domain can be acquired to be processed. These multiple frames of LDR images can correspond to different exposure times. The number of LDR images to be processed can be adjusted according to actual needs.

[0047] For example, the above-mentioned multiple frames of LDR images to be processed in the Raw domain can be 4 frames of Raw domain LDR images with different exposure times, and the exposure time ratio between adjacent frames is 1:16.

[0048] Step S102: Input multiple frames of LDR images to be processed into the image processing model to obtain the initial HDR image in the Raw domain after tone mapping output by the image processing model.

[0049] The image processing model is used to process multiple frames of LDR images in the Raw domain and output an initial HDR image in the Raw domain after tone mapping.

[0050] After obtaining the initial HDR image in the Raw domain output by the image processing model and after tone mapping, it may also include:

[0051] The initial HDR image is de-mosaiced to obtain an HDR image in the RGB domain.

[0052] For details, please refer to Figure 2 and Figure 3, Figure 2 This illustrates a conventional ISP workflow in the prior art. Raw domain data acquired by the sensor (obtained through multiple exposures) undergoes alignment and merging, demosaicing (used to convert the image from the raw domain to the RGB domain), and then tone mapping processing can be performed in the RGB domain to output the final RGB data. On the one hand, RGB domain HDR image processing techniques are not easily incorporated directly into the ISP workflow; on the other hand, devices with limited processing resources struggle to support RGB domain tone mapping processing. Figure 3 The ISP flow of this application is illustrated. The Raw domain data acquired by the sensor consists of multiple frames of LDR images to be processed in the Raw domain obtained through multiple exposures. The AI ​​HDR module inputs the multiple frames of LDR images to be processed into the image processing model to obtain the initial HDR image in the Raw domain after tone mapping output by the image processing model. Subsequently, de-mosaic processing is performed to output the HDR image in the RGB domain.

[0053] In the embodiments of this application, by acquiring multiple frames of LDR images to be processed in the Raw domain, inputting the multiple frames of LDR images to be processed into the image processing model, and obtaining the initial HDR image in the Raw domain after tone mapping output by the image processing model, the tone-mapped result can be output based on the Raw domain LDR image, which can be directly integrated into the existing ISP process to achieve more efficient HDR processing. Furthermore, the image processing model directly outputs the tone-mapped initial HDR image, which is convenient for application on devices with limited processing resources and difficulty in performing tone mapping in the RGB domain, thus helping to improve the applicability of HDR processing.

[0054] In some embodiments of this application, such as Figure 4 As shown, the image processing model's processing of multiple frames of LDR images to be processed may include steps S401 to S405.

[0055] Step S401: Extract multiple first image features corresponding to the multiple frames of LDR images to be processed.

[0056] Specifically, the LDR image to be processed can be processed through the convolutional layer of the image processing model to obtain the corresponding first image features. For example, based on existing methods, the required features can be extracted from the LDR image to be processed in 4 frames with different exposure times to obtain a high dynamic range image.

[0057] Step S402: Determine the attention result for each set of first image features based on multiple sets of first image features.

[0058] Each set of first image features includes the first image features corresponding to the target LDR image among multiple frames of LDR images to be processed, and the first image features corresponding to a frame of LDR images other than the target LDR image. The target LDR image can be any frame among the multiple frames of LDR images to be processed.

[0059] like Figure 5 The image processing model shown takes four LDR images (Exp-0 to Exp-3) as input. Each LDR image has one input channel, with Exp-1 serving as the target LDR image. The four LDR images are processed through convolutional layers to obtain multiple first image features: CNN_0, CNN_1, CNN_2, and CNN_3. Then, CNN_0, CNN_2, and CNN_3 are used to perform attention calculations with CNN_1, resulting in three attention outcomes.

[0060] Specifically, in the step of determining the attention result for each set of first image features based on multiple sets of first image features, the step of determining the attention result for a single set of first image features may include:

[0061] The first image features of the group are concatenated to obtain the concatenation result; the concatenation result is convolved and activated to obtain the activation result; the activation result is multiplied with the first image features corresponding to a frame of LDR image to be processed other than the target LDR image in the first image features of the group to obtain the attention result of the first image features.

[0062] The attention module structure of the image processing model is as follows: Figure 6 As shown, taking a set of first image features composed of CNN_0 and CNN_1 as an example, CNN_1, as the first image feature corresponding to the target LDR image to be processed, is concatenated with CNN_0 in the channel dimension to obtain the concatenated result. The concatenated result is input into a convolutional layer for convolution processing (Conv), and activated using the Sigmoid function to obtain the activation result. The activation result is multiplied by CNN_0 to obtain the attention result of this set of first image features. Similarly, the corresponding attention results can be determined for a set of first image features composed of CNN_2 and CNN_1, and for a set of first image features composed of CNN_3 and CNN_1.

[0063] Step S403: The first image features corresponding to the target LDR image to be processed and the attention results of each group of first image features are concatenated to obtain the second image features.

[0064] In some embodiments of this application, the first image features corresponding to the target LDR image to be processed and the attention results of each group of first image features can be concatenated in the channel dimension, and the concatenation result can be processed to obtain the second image features, so as to obtain the high dynamic range image.

[0065] For details, please continue to refer to [the website / information]. Figure 5 The attention result of each group of first image features and the first image feature CNN_1 corresponding to the target LDR image to be processed are concatenated in the channel dimension. After the concatenation result is processed by the convolution layer, the second image feature CNN_4 is obtained. The second image feature CNN_4 has 4 channels.

[0066] Step S404: The second image feature is split into multiple third image features that correspond one-to-one with the multiple frames of LDR images to be processed.

[0067] Specifically, the second image features can be split according to the number of LDR images to be processed, and each of the split features can be activated to obtain multiple third image features. Please refer to [link / reference]. Figure 5 The second image feature CNN_4 is split into 4 parts along the channel direction and activated by the sigmoid function respectively. This yields 4 third image features that correspond one-to-one with the LDR images Exp-0, Exp-1, Exp-2, and Exp-3 of the multi-frame LDR images to be processed. These features correspond to the proportion and other parameter features of the target pixel data of each LDR image to be processed, which are used to generate high dynamic range images.

[0068] Step S405: Multiply each third image feature with the corresponding LDR image to be processed, and add all the multiplication results together to obtain the initial HDR image.

[0069] For details, please continue to refer to [the website / information]. Figure 5 The four third image features are multiplied by their corresponding LDR images Exp-0, Exp-1, Exp-2, and Exp-3, respectively. Specifically, the third image feature corresponding to Exp-0 is multiplied by Exp-0, the third image feature corresponding to Exp-1 is multiplied by Exp-1, the third image feature corresponding to Exp-2 is multiplied by Exp-2, and the third image feature corresponding to Exp-3 is multiplied by Exp-3. All multiplication results are then summed to obtain the output result, which is the initial HDR image. The initial HDR image has one channel.

[0070] Thus, the model takes as input multiple-frame exposure LDR images in the Raw domain and outputs tone-mapped HDR images, which can be directly integrated into the existing camera's ISP workflow for more efficient HDR processing. Designing an image processing model with an attention mechanism, selecting one frame as a reference frame from the multiple-frame exposure input, and calculating the output using scalar multiplication helps solve problems such as misalignment and artifacts caused by multiple-frame exposures.

[0071] Correspondingly, for Figure 5 The image processing model shown Figure 7 This application illustrates a method for generating a training dataset for training the image processing model.

[0072] The method for generating this training dataset can be applied to electronic devices. These electronic devices can be computers, mobile phones, cameras, or other smart devices; this application does not impose any limitations on this.

[0073] Specifically, the method for generating the above training dataset may include the following steps S701 to S704.

[0074] Step S701: Obtain multi-frame sample LDR images of the target object in the Raw domain.

[0075] Step S702: Obtain a sample HDR image of the target object in the RGB domain.

[0076] Specifically, the target object can be a person, an object, a scene, or any imaging object. This application aims to emphasize that the imaging object is consistent between multi-frame sample LDR images and sample HDR images.

[0077] In some embodiments of this application, such as Figure 8 As shown, the digital camera and sensor can be set up as a binocular camera, each capturing images of the target object separately. For example, they can be fixed by a pan-tilt unit and simultaneously captured by a controller. The digital camera can be a digital single-lens reflex camera (DSLR) supporting HDR mode, and the sample HDR image can be obtained by the DSLR camera capturing the target object. Specifically, a sample HDR image can be obtained by capturing the target object in HDR mode using the DSLR camera. This sample HDR image is a high-quality image in the RGB domain, and can be a high-quality image defined according to actual needs based on the relevant parameters of the image sensor obtained in existing technology. Multiple frame sample LDR images can be acquired by the sensor. By controlling the exposure time, the sensor can obtain multiple frame sample LDR images with different exposure intensities. The number of frames can be set according to the input requirements of the image processing model; for example, the number of frames is 4, and the exposure time ratio between adjacent frames can be 1:16.

[0078] Step S703: Perform reverse demosaic processing on the sample HDR image to obtain the target image in the Raw domain.

[0079] The inverse demosaic process converts an RGB-domain image to a Raw-domain image. Inverse demosaic is performed on a sample HDR image to obtain the target image in the Raw domain.

[0080] Step S704: Generate a training dataset based on the multi-frame sample LDR images and the target image.

[0081] At this point, multiple frames of sample LDR images are used as input during training, and the target image in the Raw domain is used as the output target of the image processing model, which can generate a training dataset for training the image processing model.

[0082] In the embodiments of this application, training data in the Raw domain is obtained by processing sample HDR images in the RGB domain, which can train an image processing model specifically for Raw domain processing; the design based on sample LDR images with multiple frames of different exposure time ratios helps to solve the problem that the model is prone to being too dark in dark areas or overexposed in bright areas during model training.

[0083] Specifically, this application allows the aforementioned training dataset to contain two different data formats: real data and synthetic data.

[0084] Real data:

[0085] like Figure 9 As shown, in some embodiments of this application, performing reverse demosaic processing on the sample HDR image to obtain the target image in the Raw domain may include steps S901 to S904.

[0086] Step S901: Determine the low-quality image in the RGB domain based on the multi-frame sample LDR images.

[0087] Among them, the low-quality images have lower image quality than the sample HDR images. Specifically, this is achieved through a simple ISP process (e.g., Figure 2 The ISP processing flow shown processes multi-frame sample LDR images (LDR-Raw) to obtain low-quality L-RGB images in the RGB domain. It should also be noted that in this application, Low Dynamic Range (LDR) and High Dynamic Range (HDR) are comparative concepts, intended to describe how the solution of this application can generate high dynamic range images as defined in the prior art based on the acquired images.

[0088] Step S902: Perform dense matching on the low-quality image in the RGB domain and the sample HDR image to obtain the successfully matched image region and the correction result.

[0089] Dense matching refers to matching each pixel in a low-quality L-RGB image and a sample HDR image to obtain the corresponding pixels in the two images.

[0090] Specifically, dense matching is performed between the low-quality image and the sample HDR image to obtain successfully matched image regions. This includes: dense matching of the low-quality image and the sample HDR image to obtain a matching score image and a correction result from the sample HDR image to the low-quality image. The matching score image records the matching score at each pixel position. The correction result is used to obtain the pixel position regions corresponding to the sample HDR image and the low-quality image. The size of the matching score image and the correction result is consistent with the size of the LDR-Raw of the sample LDR image. A window of a preset size is slid across the matching score image until the average matching score within the window is greater than the average threshold. The image region within the window is then considered the successfully matched image region. In this implementation, both the preset size and the average threshold can be selected according to actual needs.

[0091] The matching score is used to characterize the degree of matching between pixels. For example, using a low-quality L-RGB image as a reference, a dense matching model can obtain a matching score image (warped-score) and a corrected result (warped-DSLR-RGB) from the sample HDR image (DSLR-RGB) to the low-quality L-RGB image (L-RGB). In this case, the dimensions of the matching score image (warped-score) and the corrected result (warped-DSLR-RGB) are consistent with the dimensions of the sample LDR image (LDR-Raw). A 256x256 window is slid across the matching score image (warped-score) in the row and column directions, and the average matching score within the window is recorded. When the average matching score is greater than the mean threshold of 0.95, the match is considered successful, and the image region within the current window is considered the successfully matched image region.

[0092] Step S903: Based on the successfully matched image regions, the correction result is cropped to obtain the first cropped image in the RGB domain.

[0093] Step S904: Perform reverse demosaic processing on the first cropped image to obtain the target image.

[0094] For a successfully matched image region, based on its location, the first cropped image Crop-DSLR-RGB in the RGB domain is obtained by cropping the corrected result Warped DSLR-RGB. The first cropped image Crop-DSLR-RGB is then subjected to reverse demosaic processing to obtain the target image Crop-target-Raw.

[0095] Accordingly, a training dataset is generated based on the multi-frame sample LDR images and the target image, which may include: cropping the multi-frame sample LDR images according to the successfully matched image regions to obtain a second cropped image in the Raw domain; and using the second cropped image and the target image as training data for the training dataset.

[0096] That is, for a successfully matched image region, it is cropped (ROI-Crop) on multiple frames of LDR-Raw according to its position, to obtain multiple frames of second cropped images Crop-LDR Raws in the Raw domain.

[0097] Therefore, combining multiple frames of the second cropped image (Crop-LDR Raws) and the target image (Crop-target-Raw) yields a set of raw domain training data, LDR-Target Raw Paire. This instance-based processing can be referenced accordingly. Figure 10 The process is shown below.

[0098] Considering that the bit width of the result of the ISP merging operation is usually higher than that of the training data, in order to facilitate the direct integration of the image processing model into the original ISP process, the multi-frame sample LDR image LDR-Raw can be saved in 10-bit format, and the target image Crop-target-Raw data can be saved in 12-bit format.

[0099] The real-data processing method involves processing images actually captured by sensors and digital SLR cameras to obtain training data. This allows the trained image processing model to be applied to real-world scenarios, improving its generalization ability. Furthermore, the target image obtained from the reverse desacrifice processing of sample HDR images is a tone-mapped image. This method can obtain a dataset of tone-mapped HDR images from LDR to HDR in the raw domain.

[0100] Synthetic data:

[0101] like Figure 11 As shown, in some embodiments of this application, obtaining a sample HDR image of the target object in the RGB domain may include steps S1001 to S1003.

[0102] Step S1001: Obtain single-frame HDR data from the HDR video.

[0103] Specifically, continuous HDR video can be parsed into single-frame HDR data according to video standards.

[0104] Step S1002: Convert the single-frame HDR data into continuous RGB values ​​between 0 and 1.

[0105] For a single frame of HDR data, 10-bit data can be converted into continuous RGB values ​​between 0 and 1.

[0106] Specifically, for single-frame HDR data in HLG (Hybrid Log-Gamma) format, the conversion formula is:

[0107] rgb = rgb ÷ 1023 ①;

[0108]

[0109] For single-frame HDR data in PQ (Perceptual Quantizer) format, the conversion formula is:

[0110] rgb = (rgb ÷ 1023) 1 / 0.78.8437 -0.8359375③;

[0111]

[0112] Step S1003: Perform tone mapping processing based on continuous RGB values ​​to obtain sample HDR images.

[0113] Specifically, existing tone mapping algorithms can be used to linearize RGB continuous values. rgb The sample HDR image is converted into an RGB domain image and then tone-mapped. In step S703, inverse demosaic processing is performed on the sample HDR image to obtain the target image Target-Raw in the Raw domain. The target image Target-Raw can be saved in 12-bit format.

[0114] Accordingly, in some embodiments of this application, acquiring multi-frame sample LDR images of the target object in the Raw domain may include:

[0115] Based on the continuous RGB values, determine multi-frame sample RGB images with different exposure ratios; perform reverse demosaic processing on the multi-frame sample RGB images to obtain multi-frame sample LDR images in the Raw domain.

[0116] Specifically, based on the obtained RGB continuous values ​​linear rgb By using an exposure ratio of 1:16, multiple frames of sample RGB images with different exposure ratios can be obtained sequentially.

[0117] Taking 4 frames of exposure as an example, the calculation formula is:

[0118] multi-exposure rgbs = {linear rgb ×γ×16 -i, i = 0, 1, 2, 3}⑥;

[0119] Where i represents the number of the 4 exposure data frames, i=0 represents the frame with the longest exposure time, and i=3 represents the frame with the shortest exposure time. γ is the adjustment factor, used to adjust the range of HDR data, and needs to maintain a proportional relationship with the target data.

[0120] Specifically, the formula for calculating the adjustment ratio is as follows:

[0121]

[0122] Here, mean(Target) represents the mean of the target data. Cropping multiple frames of sample RGB images with different exposure ratios using the clip(0,1023) operation ensures that each frame's RGB image ranges from 0 to 1023. Reverse de-mosaicing of the processed multi-frame sample RGB images (using the same reverse de-mosaicing process as when acquiring real data) yields the raw domain multi-frame sample LDR image LDR-Raws, which can be saved as a 10-bit file.

[0123] At this point, a training dataset is generated based on multiple sample LDR images and the target image. This may include cropping the target image and the sample LDR images at the same location and using the cropped images as training data for the training dataset.

[0124] For example, a 256x256 window can be used to crop the target image (Target-Raw) and sample LDR images (LDR-Raws) at the same location to obtain a set of training data.

[0125] In this way, training data can be obtained from HDR videos as a supplement to real data.

[0126] for Figure 5 The image processing model shown Figure 12 This application illustrates a model training method for obtaining the image processing model. This model training method can be applied to electronic devices.

[0127] Specifically, the model training method may include steps S1101 to S1102.

[0128] Step S1101: Obtain the training dataset.

[0129] For instructions on how to obtain the training dataset, please refer to [link / reference]. Figures 7-11 The description is omitted here.

[0130] Step S1102: Input the multi-frame sample LDR images from the training dataset into the model to be trained, and train the model with the goal of minimizing the loss value of the model to be trained, so as to obtain the image processing model.

[0131] The loss value is related to the difference between the output of the model being trained and the target image. Specifically, the difference between the output and the target-Raw image can be constrained by the mean squared error (MSE) loss and L1 loss. Through iterative training, the loss value is minimized (i.e., the difference is minimized), enabling the image processing model to accurately output the tone-mapped HDR image in the Raw domain.

[0132] In some embodiments of this application, to accelerate model convergence, the sample LDR images (LDR-Raws) and target images (Target-Raws) in the training dataset can be normalized to between 0 and 1 before training. Specifically, the sample LDR images can be divided by 1023 to normalize to between 0 and 1, and the target images (Target-Raws) can be divided by 4095 to normalize to between 0 and 1.

[0133] Understandably, in the training dataset, the target image (Target-Raw) is obtained from high-quality HDR sample images captured by a DSLR camera, or from HDR videos after inverse desamicing. During training, the model learns the mapping relationship between LDR and HDR images in the Raw domain, and the model output during inference is equivalent to the result after tone mapping. Therefore, the trained image processing model can be directly embedded between the raw domain input and desamicing processing in the ISP process, eliminating the tone mapping process in the conventional ISP process and achieving more efficient HDR image output.

[0134] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders.

[0135] like Figure 13 The diagram shown is a structural schematic of an HDR image generation device 1200 provided in an embodiment of this application. The HDR image generation device 1200 is disposed on an image sensor.

[0136] Specifically, the HDR image generation device 1200 may include:

[0137] Acquisition unit 1201 is used to acquire multiple frames of LDR images to be processed in the Raw domain;

[0138] The image generation unit 1202 is used to input the multi-frame LDR images to be processed into the image processing model to obtain the initial HDR image in the Raw domain after tone mapping output by the image processing model.

[0139] It should be noted that, for the sake of convenience and brevity, the specific working process of the HDR image generation device 1200 described above can be found in [reference needed]. Figures 1 to 12 The corresponding process of the method will not be described in detail here.

[0140] In some embodiments of this application, the image generation unit 1202 is specifically used for: extracting multiple first image features corresponding to the multiple frames of LDR images to be processed; determining the attention result of each set of first image features based on the multiple sets of first image features, wherein each set of first image features includes the first image feature corresponding to the target LDR image in the multiple frames of LDR images to be processed, and the first image feature corresponding to a frame of LDR images other than the target LDR image to be processed; concatenating the first image feature corresponding to the target LDR image to be processed and the attention result of each set of first image features to obtain a second image feature; splitting the second image feature into multiple third image features corresponding to the multiple first image features; multiplying each third image feature with the corresponding first image feature, and adding all the multiplication results to obtain the initial HDR image.

[0141] In some embodiments of this application, the image generation unit 1202 is specifically used to: stitch together a single set of first image features to obtain a stitching result; perform convolution processing on the stitching result and activate it to obtain an activation result; multiply the activation result with the first image features corresponding to a frame of LDR image to be processed other than the target LDR image in the set of first image features to obtain an attention result for the set of first image features.

[0142] In some embodiments of this application, the image generation unit 1202 is further configured to: perform demosaic processing on the initial HDR image to obtain an HDR image in the RGB domain.

[0143] like Figure 14 The diagram shown is a schematic representation of a training dataset generation device 1300 provided in an embodiment of this application. The training dataset generation device 1300 is configured on an electronic device. This electronic device can be a computer, mobile phone, or other smart device, and this application does not limit its use.

[0144] Specifically, the training dataset generation device 1300 may include:

[0145] The first acquisition unit 1301 is used to acquire multi-frame sample LDR images of the target object in the Raw domain;

[0146] The second acquisition unit 1302 is used to acquire a sample HDR image of the target object in the RGB domain;

[0147] The target image acquisition unit 1303 is used to perform reverse demosaicing on the sample HDR image to obtain the target image in the Raw domain.

[0148] The training dataset generation unit 1304 is used to generate a training dataset based on the multi-frame sample LDR images and the target image.

[0149] In some embodiments of this application, the sample HDR images are obtained by taking pictures of the target object with a digital SLR camera that supports HDR mode, and the multi-frame sample LDR images are obtained by a sensor.

[0150] In some embodiments of this application, the target image acquisition unit 1303 is specifically used for: determining a low-quality image in the RGB domain based on the multi-frame sample LDR images, wherein the image quality of the low-quality image is lower than that of the sample HDR image; performing dense matching on the low-quality image and the sample HDR image to obtain a successfully matched image region and a correction result; cropping the correction result based on the successfully matched image region to obtain a first cropped image in the RGB domain; and performing reverse demosaic processing on the first cropped image to obtain the target image.

[0151] In some embodiments of this application, the training dataset generation unit 1304 is specifically used to: crop the multi-frame sample LDR image according to the successfully matched image region to obtain a second cropped image in the Raw domain; and use the second cropped image and the target image as training data for the training dataset.

[0152] In some embodiments of this application, the target image acquisition unit 1303 is specifically used for: performing dense matching on the low-quality image and the sample HDR image to obtain a matching score image from the sample HDR image to the low-quality image, wherein the matching score image records the matching score at each pixel position; sliding a window of a preset size on the matching score image until the average matching score within the window is greater than the average threshold, and taking the image region within the window as the successfully matched image region.

[0153] In some embodiments of this application, the second acquisition unit 1302 is specifically used to: acquire single-frame HDR data in an HDR video; convert the single-frame HDR data into continuous RGB values ​​between 0 and 1; and perform tone mapping processing based on the continuous RGB values ​​to obtain the sample HDR image.

[0154] In some embodiments of this application, the first acquisition unit 1301 is specifically used to: determine multi-frame sample RGB images with different exposure ratios based on the RGB continuous values; and perform reverse demosaic processing on the multi-frame sample RGB images to obtain the multi-frame sample LDR images in the Raw domain.

[0155] In some embodiments of this application, the training dataset generation unit 1304 is specifically used to: crop the target image and the sample LDR image at the same position, and use the cropped image as the training data of the training dataset.

[0156] It should be noted that, for the sake of convenience and brevity, the specific working process of the above-mentioned training dataset generation device 1300 can be found in [reference needed]. Figures 1 to 12 The corresponding process of the method will not be described in detail here.

[0157] like Figure 15 The diagram shown is a structural schematic of a model training device 1400 provided in an embodiment of this application. The model training device 1400 is configured on an electronic device.

[0158] Specifically, the model training device 1400 may include:

[0159] Training set acquisition unit 1401 is used to acquire the training dataset;

[0160] The training unit 1402 is used to input the multi-frame sample LDR images in the training dataset into the model to be trained, and to train the model with the goal of minimizing the loss value of the model to be trained, so as to obtain an image processing model; wherein the loss value and the output result of the model to be trained are related to the difference between the target image.

[0161] It should be noted that, for the sake of convenience and brevity, the specific working process of the aforementioned model training device 1400 can be found in the following reference: Figures 1 to 11 The corresponding process of the method will not be described in detail here.

[0162] like Figure 16 As shown, this application also provides an image sensor 15, including:

[0163] The data acquisition unit 151 is used for processing multiple frames of LDR images in the Raw domain. The data acquisition unit 151 may include a pixel array and an analog-to-digital converter to convert the acquired optical signals into digital signals.

[0164] Image processing unit 152 (ISP processor), connected to data acquisition unit 151, is used to input multiple frames of LDR images to be processed into image processing model to obtain an initial HDR image in the Raw domain output by image processing model after tone mapping.

[0165] Furthermore, the image processing unit 152 can perform de-mosaic processing on the initial HDR image to obtain an HDR image in the RGB domain.

[0166] Specifically, light shines on the object being imaged and is reflected, then converged by the lens of the image sensor to the image sensor 15. The image sensor 15 performs photoelectric conversion, and the image is then converted by an analog-to-digital converter (ADC) to obtain a raw image. The raw image is the original data captured by the image sensor 15 and converted into a digital signal.

[0167] The memory inside the image processing unit 152 stores a computer program that can run the HDR image generation method described above, process the RAW image from the image sensor 15, and convert the RAW image into an HDR image in the RGB domain.

[0168] It should be noted that, for the sake of convenience and brevity, the structure of the above-mentioned electronic device can also be referred to the specific description of the structure in the method embodiment, which will not be repeated here.

[0169] like Figure 17 The diagram shown is a schematic representation of an electronic device provided in an embodiment of this application. Specifically, the electronic device 16 may include: a processor 16, a memory 161, and a computer program 162 stored in the memory 161 and executable on the processor 160.

[0170] When processor 160 executes computer program 162, it implements the steps in the above-described embodiments of the methods for generating training datasets, for example... Figure 7 Steps S701 to S704 are shown. Alternatively, when the processor 160 executes the computer program 162, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 13 The functions of the first acquisition unit 1301, the second acquisition unit 1302, the target image acquisition unit 1303, and the training dataset generation unit 1304 are shown.

[0171] Alternatively, when processor 160 executes computer program 162, it implements the steps in the various model training method embodiments described above, for example... Figure 12 The steps S1101 to S1102 are shown. Alternatively, when the processor 160 executes the computer program 162, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 15 The functions of the training set acquisition unit 1401 and the training unit 1402 shown are illustrated.

[0172] The computer program can be divided into one or more modules / units, which are stored in the memory 161 and executed by the processor 160 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0173] The electronic device may include, but is not limited to, a processor 160 and a memory 161. Those skilled in the art will understand that... Figure 17 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0174] The processor 160 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0175] The memory 161 can be an internal storage unit of the electronic device, such as a hard drive or memory. The memory 161 can also be an external storage device of the electronic device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 161 can include both internal and external storage units. The memory 161 is used to store the computer program and other programs and data required by the electronic device. The memory 161 can also be used to temporarily store data that has been output or will be output.

[0176] It should be noted that, for the sake of convenience and brevity, the structure of the above-mentioned electronic device can also be referred to the specific description of the structure in the method embodiment, which will not be repeated here.

[0177] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0178] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0179] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for various specific applications, but such implementations should not be considered beyond the scope of this application.

[0180] In the embodiments provided in this application, it should be understood that the disclosed devices / image sensors / electronic devices and methods can be implemented in other ways. For example, the device / image sensor / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0183] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0184] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for generating HDR images, characterized in that, include: Acquire multiple frames of LDR images to be processed in the Raw domain; The multiple LDR images to be processed are input into the image processing model to obtain the initial HDR image in the Raw domain after tone mapping, which is output by the image processing model.

2. The method for generating HDR images as described in claim 1, characterized in that, The image processing model's processing of the multiple frames of LDR images to be processed includes: Extract multiple first image features corresponding to the multiple frames of LDR images to be processed; Based on multiple sets of first image features, determine the attention result for each set of first image features, where: Each set of first image features includes the first image features corresponding to the target LDR image in the multi-frame LDR images to be processed, and the first image features corresponding to a frame of LDR images to be processed other than the target LDR image. The first image features corresponding to the target LDR image to be processed and the attention results of each group of first image features are concatenated to obtain the second image features; The second image feature is split into multiple third image features that correspond one-to-one with the multiple frames of LDR images to be processed; Each of the third image features is multiplied by the corresponding LDR image to be processed, and all multiplication results are summed to obtain the initial HDR image.

3. The method for generating HDR images as described in claim 2, characterized in that, In the step of determining the attention result of each group of first image features based on multiple groups of first image features, the step of determining the attention result of a single group of first image features includes: The first set of image features are stitched together to obtain the stitched result; The splicing result is then subjected to convolution and activation to obtain the activation result; The activation result is multiplied by the first image feature corresponding to a frame of LDR image other than the target LDR image in the first image feature group to obtain the attention result of the first image feature group.

4. The method for generating HDR images as described in any one of claims 1-3, characterized in that, The multiple frames of LDR images to be processed correspond to different exposure times; And / or, after obtaining the initial HDR image in the Raw domain output by the image processing model and after tone mapping, the method further includes: The initial HDR image is de-mosaiced to obtain an HDR image in the RGB domain.

5. A method for generating a training dataset, characterized in that, The training dataset is used to train the image processing model according to any one of claims 1-4; The method for generating the training dataset includes: Acquire multi-frame sample LDR images of the target object in the Raw domain; Obtain a sample HDR image of the target object in the RGB domain; The target image in the Raw domain is obtained by performing reverse demosaic processing on the sample HDR image. A training dataset is generated based on the multi-frame sample LDR images and the target image.

6. The method for generating the training dataset as described in claim 5, characterized in that, The sample HDR images were captured by a digital SLR camera that supports HDR mode on the target object, while the multi-frame sample LDR images were acquired by a sensor. And / or, The step of performing inverse demosaic processing on the sample HDR image to obtain the target image in the Raw domain includes: Based on the multi-frame sample LDR images, a low-quality image in the RGB domain is determined, wherein the image quality of the low-quality image is lower than that of the sample HDR image; Dense matching is performed on the low-quality image and the sample HDR image to obtain the successfully matched image regions and correction results; Based on the successfully matched image region, the correction result is cropped to obtain the first cropped image in the RGB domain; The first cropped image is subjected to reverse demosaic processing to obtain the target image.

7. The method for generating the training dataset as described in claim 6, characterized in that, The step of generating a training dataset based on the multi-frame sample LDR images and the target image includes: Based on the successfully matched image regions, the multi-frame sample LDR images are cropped to obtain a second cropped image in the Raw domain; The second cropped image and the target image are used as training data for the training dataset.

8. The method for generating the training dataset as described in claim 6, characterized in that, Dense matching is performed between the low-quality image and the sample HDR image to obtain successfully matched image regions, including: Dense matching is performed on the low-quality image and the sample HDR image to obtain a matching score image from the sample HDR image to the low-quality image. The matching score image records the matching score at each pixel position. Slide a window of a preset size across the matching score image until the average matching score within the window is greater than the average threshold, then the image region within the window is taken as the successfully matched image region.

9. The method for generating the training dataset as described in claim 5, characterized in that, The step of obtaining a sample HDR image of the target object in the RGB domain includes: Obtain single-frame HDR data from an HDR video; The single-frame HDR data is converted into continuous RGB values ​​between 0 and 1; The sample HDR image is obtained by performing tone mapping processing based on the continuous RGB values.

10. The method for generating the training dataset as described in claim 9, characterized in that, The acquisition of multi-frame sample LDR images of the target object in the Raw domain includes: Based on the continuous RGB values, determine multi-frame sample RGB images with different exposure time ratios; The multi-frame sample RGB images are subjected to inverse demosaic processing to obtain the multi-frame sample LDR images in the Raw domain.

11. The method for generating the training dataset as described in claim 9, characterized in that, The step of generating a training dataset based on the multi-frame sample LDR images and the target image includes: The target image and the sample LDR image are cropped at the same location, and the cropped image is used as the training data of the training dataset.

12. A model training method, characterized in that, include: Obtain a training dataset, which is obtained by the method for generating a training dataset according to any one of claims 5-11; The LDR images of multiple frames in the training dataset are input into the model to be trained, and the model is trained with the goal of minimizing the loss value of the model to be trained, so as to obtain the image processing model; wherein the loss value and the output result of the model to be trained are related to the difference between the target image.

13. An image sensor, characterized in that, include: The data acquisition unit is used to acquire multiple frames of LDR images to be processed in the Raw domain; An image processing unit, connected to the data acquisition unit, is used to input the multi-frame LDR images to be processed into an image processing model to obtain an initial HDR image in the Raw domain after tone mapping output by the image processing model.

Images