Image processing method and system, image signal processor and electronic equipment

Abstract

The embodiment of the invention discloses an image processing method and system, an image signal processor, electronic equipment and a storage medium. The method comprises the following steps: performing detail enhancement processing on original image data acquired by an image sensor based on a first image processing model to obtain first image data; performing image correction processing on the first image data to obtain second image data; and brightness adjustment processing is performed on the second image data based on a second image processing model to obtain target image data, the first image processing model is trained based on preset original data and RGB data in advance, and the second image processing model is trained based on preset original dark image and target bright image data in advance.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and system, an image signal processor, an electronic device and a storage medium. Background Art

[0002] Nowadays, with the continuous development of intelligent technology, various application scenarios such as intelligent security, autonomous driving, and underwater detection have emerged. How to capture high-quality and clear images in low-light environments has become a universal demand.

[0003] However, due to insufficient light in low-light environments, when the brightness of the scene is low to a certain level, the images taken with ordinary cameras or camcorders will have problems such as dark images, loss of details, high noise, color degradation, etc., which brings great difficulties to subsequent image analysis and processing.

[0004] At present, a variety of low-light image enhancement algorithm solutions have been proposed, which can be roughly divided into two categories: traditional image enhancement algorithms and deep learning-based image enhancement algorithms. Traditional image enhancement algorithms mainly improve image quality by performing image noise reduction, contrast enhancement, digital gain brightening and other processing. However, such traditional processing methods often have problems such as unclean noise removal, loss of image information, and pseudo texture. Image enhancement algorithms based on deep learning have significant advantages over traditional image enhancement algorithms. Such algorithms usually use network models to further enhance the RGB (Red Green Blue) images output by the front-end ISP (Image Signal Processing). However, such algorithms are used as back-end processing algorithms to post-process images processed by the ISP module. Therefore, the effect of such algorithms is not only limited by the front-end ISP module, but also increases the complexity of the entire image processing process.

[0005] In summary, how to enhance low-light images and improve their quality has become a technical problem that needs to be solved. Summary of the invention

[0006] The technical solution provided by this application at least partially solves the above technical problems.

[0007] According to a first aspect of the present application, there is provided an image processing method for an image signal processor, the method comprising: performing detail enhancement processing on original image data collected by an image sensor based on a first image processing model to obtain first image data; performing image correction processing on the first image data to obtain second image data; and performing brightness adjustment processing on the second image data based on a second image processing model to obtain target image data, wherein the first image processing model is pre-trained based on preset original data and RGB data, and the second image processing model is pre-trained based on preset original dark image and target bright image data.

[0008] In some embodiments, the above method also includes: performing degradation processing on the acquired high-definition image frame to obtain degraded original data; using a pre-calibrated noise model to superimpose noise on the degraded original data to obtain noisy original data; performing sharpening processing and degradation processing on the high-definition image frame to obtain degraded RGB data; using the noisy original data and the degraded RGB data as the first training data pair to construct a first training data set to train the first image processing model.

[0009] In some implementations, performing degradation processing on the high-definition image frame to obtain degraded original data includes: performing degradation processing on the high-definition image frame, and extracting data according to a set Bayer format to obtain the degraded original data.

[0010] In some embodiments, the degradation process includes the following processes: inverse gamma correction, inverse dynamic range compression, inverse color correction, and inverse white balance.

[0011] In some embodiments, the pre-calibrated noise model is obtained as follows: data of preset ISO values ​​are obtained in a darkroom experimental environment, wherein two groups of data including normal exposure data and underexposure data are respectively captured at each preset ISO value; the mean and variance of each pixel point at each preset ISO value are statistically calculated based on the captured data to obtain the noise intensity parameter of each preset ISO value, and a linear noise model is fitted as the noise model.

[0012] In some embodiments, the above method also includes: converting the original dark image to the HSV color space and performing inverse gamma correction on the brightness V channel of the HSV color space; converting the target bright image to the HSV color space and extracting the hue H channel and saturation S channel of the HSV color space; merging the H channel and S channel of the target bright image with the inverse gamma-corrected V channel of the original dark image, and converting them to the RGB color space to obtain an updated dark image; using the updated dark image and the target bright image as a second training data pair to construct a second training data set to train the second image processing model.

[0013] In some implementations, the detail enhancement processing includes at least one of demosaicing, noise reduction, and sharpening.

[0014] In some embodiments, the brightness adjustment process includes at least one of gamma correction and dynamic range compression.

[0015] In some implementations, the image correction process includes at least one of black level correction, automatic white balance, lens shading correction, and color correction.

[0016] According to a second aspect of the present application, an image signal processor is provided, which includes: a first image processing model, used to perform detail enhancement processing on original image data collected by an image sensor to obtain first image data; an image correction module, used to perform image correction processing on the first image data to obtain second image data; and a second image processing model, used to perform brightness adjustment processing on the second image data to obtain target image data, wherein the first image processing model is pre-trained based on preset original data and RGB data, and the second image processing model is pre-trained based on preset original dark image and target bright image data.

[0017] In some embodiments, the detail enhancement processing includes at least one of demosaicing, noise reduction, and sharpening; the image correction processing includes at least one of black level correction, automatic white balance, lens shading correction, and color correction; the brightness adjustment processing includes at least one of gamma correction and dynamic range compression.

[0018] In some embodiments, the image signal processor further includes: an input module for receiving raw image data collected by the image sensor; and an output module for obtaining standard RGB image data according to the target image data and outputting the data.

[0019] In some embodiments, the first image processing model and the second image processing model include a deep convolutional neural network model or a recurrent neural network model.

[0020] According to a third aspect of the present application, an image processing system is provided, comprising an image sensor for acquiring raw image data to be processed, and also comprising the above-mentioned image signal processor.

[0021] According to the fourth aspect of the present application, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the first aspect of the present application.

[0022] According to the fifth aspect of the present application, a computer-readable storage medium is also provided, on which machine-executable instructions are stored. When the machine-executable instructions are executed, the machine executes the method according to the first aspect of the present application.

[0023] According to the implementation manner of the present application, an image signal processor (ISP) is provided. By adopting two pre-trained neural network models to replace some traditional ISP algorithm modules, not only the ISP link is simplified, but also the problems of color distortion and increased noise in low-light scenes are better solved, thereby obtaining a better enhancement effect.

[0024] The image processing algorithm according to the implementation mode of the present application can be applied to ISP, combining the network enhancement algorithm based on deep learning with the traditional ISP algorithm. The powerful learning ability of the AI ​​(Artificial Intelligence) algorithm makes up for the shortcomings of the traditional algorithm in enhancement effect, improves the defects of noise residue, color distortion, poor brightness and contrast in the traditional scheme, solves the problem that the enhancement effect of the back-end enhancement algorithm scheme is limited by the front-end ISP module, and achieves better image effect.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Other features, purposes and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present application. In the drawings:

[0027] Figure 1 is a schematic diagram of an image processing flow of a front-end ISP according to the related art;

[0028] Figure 2 is a flowchart of an image processing method provided according to an embodiment of the present application;

[0029] Figure 3 is a schematic diagram of the structure of an ISP provided according to an embodiment of the present application;

[0030] Figure 4 is a schematic diagram of a process for constructing a first training data set according to an implementation manner of the present application;

[0031] Figure 5 is a schematic diagram of a process for constructing a second training data set according to an embodiment of the present application;

[0032] Figure 6is a schematic diagram of the composition of an image signal processor according to an embodiment of the present application; and

[0033] Figure 7 It is a schematic diagram of the composition of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to better understand the present application, a more detailed description will be made of various aspects of the present application with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application, and are not intended to limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0035] It should also be understood that expressions such as "include", "including", "have", "contain" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features rather than just the individual elements in the list. In addition, when describing embodiments of the present application, "may" is used to mean "one or more embodiments of the present application". And, the term "exemplary" is intended to refer to an example or illustration.

[0036] In the description of the embodiments of the present application, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, the order in which the steps are described does not necessarily refer to the order in which the steps are performed, unless it can be determined from the context that the steps are performed sequentially.

[0037] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. Any one or more embodiments or elements thereof, or the arrangement or combination of some or all embodiments or elements thereof described herein are merely exemplary and are not intended to limit the scope of the present application.

[0038] The exemplary embodiments of the present application are described below in conjunction with the accompanying drawings. Various details of the embodiments of the present application are included to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application.

[0039] As mentioned above, current deep learning-based image enhancement algorithms usually use network models to further enhance the displayable color images, such as RGB images, output by the front-end ISP (image signal processor).

[0040] ISP is generally used to perform front-end processing on the Raw image data collected by the Sensor. Raw image data is the digital image output by the Sensor without any processing, compression or noise reduction, and the original data is retained for subsequent processing. Figure 1 As shown, according to the related technology, the traditional ISP can perform black level correction, automatic white balance, green imbalance correction, Bayer noise reduction, lens shading correction, demosaicing, color correction, Gamma correction, dynamic range compression and sharpening on the Raw image data collected by the Sensor, and then convert the Raw image data into displayable standard RGB image data (sRGB) for output.

[0041] The image enhancement algorithm based on deep learning uses a network model to perform back-end processing on the RGB images output by the ISP. Among them, there are two common types: methods based on image decomposition and methods that use direct processing using network models.

[0042] Methods based on image decomposition usually require three different network models and use the following specific steps: 1) Based on the Retinex theory, a decomposition network is used to decompose the input image into an illumination map and a reflectance map. The illumination map is used to represent the lighting information, and the reflectance map is used to represent the inherent color properties of the object; 2) The reflectance map is processed by a reflection restoration network, including noise reduction, color restoration, etc.; 3) The illumination adjustment network is used to brighten the illumination map; 4) The processed illumination map and reflectance map are synthesized to output an enhanced image. Although this type of method can effectively enhance the image, it has many defects: the requirements for the decomposition network are high, and the quality of image decomposition directly affects the processing effect of the subsequent reflection restoration network and illumination adjustment network; the processing flow is too complicated, and the use of multiple network models greatly increases the algorithm overhead, and the feasibility of implementation is low; the reflection restoration network is difficult to deal with the two major problems of noise and color at the same time, and the processing effect is insufficient.

[0043] The method of direct processing using a network model is relatively simple and exists in two forms: a) directly applying the network model to predict the enhanced image; b) combining the network model with the Retinex theory, applying the network model to predict the illumination map, and then calculating the reflectance map as the enhanced image. Although this type of method has significantly lower overhead than the former, it also has some defects, such as not considering the noise problem, and the enhancement result has obvious noise problems; and it is disconnected from the processing of the front-end ISP, and the ISP processing will cause the loss of image data details, color degradation and other problems, resulting in the enhancement effect of the back-end algorithm has great limitations. In other words, the processing algorithm of the traditional ISP will cause the loss of some detail information of the original image data. Under the premise of the loss of detail information, the back-end image enhancement processing algorithm cannot be optimized.

[0044] In order to solve the above problems, an image processing method 200 is provided according to an exemplary embodiment of the present application, which can be applied to ISP to achieve enhanced processing of low-light images. Figure 2 As shown, the method 200 as a whole may include:

[0045] Step S1, performing detail enhancement processing on original image data collected by an image sensor based on a first image processing model to obtain first image data, wherein the detail enhancement processing at least includes de-mosaicing.

[0046] Exemplarily, the detail enhancement processing implemented based on the first image processing model may include noise reduction, demosaicing and sharpening.

[0047] Step S2, performing image correction processing on the first image data to obtain second image data, wherein the image correction processing may include but is not limited to one of the following processing or any combination thereof: black level correction, automatic white balance, lens shading correction, and color correction.

[0048] Step S3: Perform brightness adjustment processing on the second image data based on the second image processing model to obtain target image data, wherein the brightness adjustment processing at least includes gamma correction.

[0049] Exemplarily, the brightness adjustment process implemented based on the second image processing model may include dynamic range compression and gamma correction.

[0050] The first image processing model is pre-trained based on preset original data and RGB data, and the second image processing model is pre-trained based on preset original dark image and target bright image data.

[0051] According to the image processing method of the implementation mode of the present application, a processing algorithm based on deep learning is integrated into the front-end ISP, and a pre-trained neural network model is used to implement image detail enhancement processing and brightness adjustment processing. On the one hand, the data processing flow is simplified, and on the other hand, since the Raw image data output by the Sensor is directly processed, the color degradation and noise problems in image processing are better solved, and a better enhancement effect is obtained, thereby improving the imaging quality in low-light scenes.

[0052] According to an exemplary embodiment of the present application, an image signal processor ISP is also provided. Figure 3 The ISP architecture 10 for low-light scenes is shown, and the ISP architecture 10 mainly includes a first image processing model 110, an image correction module 120, and a second image processing model 130. The input of the ISP structure 10 is the Raw image data collected by the Sensor, and two image processing models are used to replace a part of the traditional processing modules, wherein the first image processing model 110 can realize the image detail enhancement processing function, such as de-mosaicing and other functions; the second image processing model 130 can realize the image brightness and / or contrast adjustment function, such as Gamma correction and other functions. The image correction module 120 can use the original ISP processing module / unit, for example, including but not limited to a black level correction unit, an automatic white balance unit, a lens shading correction unit, and a color correction unit.

[0053] The features, principles and other aspects of the present application will be described in detail below in conjunction with specific embodiments.

[0054] Embodiment 1

[0055] This embodiment provides an image processing method, which can be applied to an improved ISP architecture to perform image signal processing on Raw image data collected by a sensor, and to enhance low-illumination images by integrating a deep learning-based image processing algorithm into the ISP processing flow.

[0056] Based on Figure 3 As an example, the ISP architecture shown in the figure is combined with Figure 2 , Figure 4 and Figure 5 As shown, the image processing method of this embodiment may include the following main steps:

[0057] Step S1, performing detail enhancement processing on the original image data based on the first image processing model to obtain first image data. Step S1 may include the following sub-steps:

[0058] Step S101, inputting the Raw image data collected and output by the Sensor into the first image processing model;

[0059] Step S102: Use the first image processing model to perform detail enhancement processing on the Raw image data to obtain first image data. Exemplarily, the detail enhancement processing performed by the first image processing model on the Raw image data includes noise reduction (such as Bayer noise reduction and green imbalance correction), demosaicing and sharpening.

[0060] Since noise reduction, demosaicing and sharpening are all pixel-level processing and focus on image detail texture, a network model is used to process these three tasks simultaneously. That is, the first image processing model can replace the green imbalance correction, Bayer noise reduction, demosaicing and sharpening processing in the traditional ISP processing flow.

[0061] Step S2, performing image correction processing on the first image data to obtain second image data. Exemplarily, the image correction processing in this embodiment includes the following processing in sequence: black level correction, automatic white balance, lens shading correction, and color correction.

[0062] It should be noted that in the traditional ISP link, black level correction, automatic white balance and lens shading correction are all processed in the Raw domain, and color correction is performed in the RGB domain. The improved ISP architecture provided in this application can process all four parts in the RGB domain. Exemplarily, step S2 may include the following sub-steps:

[0063] Step S201: black level correction processing, subtracting the black levels of the three channels R, G, and B.

[0064] Step S202, automatic white balance processing, that is, finding the white point in the image and calculating the R channel gain (R_Gain) and the B channel gain (B_Gain). Usually, the G channel gain (G_Gain) defaults to 1, and then multiplying each channel by the corresponding gain value to obtain the image after white balance processing.

[0065] Since the image in low-light scenes is generally darker, in order to improve the accuracy of the automatic white balance algorithm, the input image needs to be linearly brightened. In addition, in order to reduce the computational overhead, the input image size can be adjusted to 512*512 when calculating the gain.

[0066] Step S203: lens shading correction processing, that is, eliminating the brightness unevenness problem caused by the lens.

[0067] Step S204: color correction processing, namely, correcting the image color.

[0068] After the above steps S201 - S204 , the second image data is obtained.

[0069] Step S3, performing brightness adjustment processing on the second image data based on the second image processing model to obtain target image data. Step S3 may include the following sub-steps:

[0070] Step S301, inputting the second image data into the second image processing model;

[0071] Step S302: Use the second image processing model to perform brightness adjustment processing on the second image data to obtain target image data. Exemplarily, the brightness adjustment processing implemented based on the second image processing model may include dynamic range compression and gamma correction.

[0072] Among them, the common point between dynamic range compression and gamma correction is that they are both used for brightness and contrast adjustment on the scale of image area, so they can also be processed simultaneously using one network model. That is, the second image processing model can replace the gamma correction and dynamic range compression processing in the traditional ISP processing flow.

[0073] The target image data obtained after step S3 is RGB image data, and standard RGB image data can be obtained based on the target image data and used as the output of the entire ISP. The standard RGB image data can be directly output to a display screen for display, or stored in a memory as a processed image.

[0074] Furthermore, the first image processing model and the second image processing model can construct a deep learning neural network model, such as a deep convolutional neural network (CNN) model or a recurrent neural network model (RNN), and construct a training data set based on preset training data to train the image processing network model and learn the representation information of the low-light image.

[0075] As an exemplary implementation, a low-quality sample image and a high-quality target image may be used as a training data pair to construct a data training set, and the constructed training data set and a preset loss function are used to train the image processing model. Exemplarily, training the image processing model using the training data set and the preset loss function may include:

[0076] Input sample data into the image processing model, and use the output of the image processing model as the predicted value;

[0077] Determining a loss value of the predicted value relative to the target image according to a preset loss function; and

[0078] The weight parameters of the image processing model are modified according to the loss value until the loss value is less than the preset target loss value. The image processing model converges and the trained image processing model is obtained.

[0079] Therefore, the key to constructing an image processing model is how to obtain a training data set for network learning. The image processing method of this embodiment may also include constructing a training data set, and pre-training the image processing model based on the constructed training data set to obtain a trained image processing model.

[0080] Exemplarily, the first image processing model can be trained based on a constructed first training data set. For example, Raw data and RGB data can be obtained according to a high-definition image frame as a first training data pair to construct the first training data set.

[0081] like Figure 4 As shown, the process 400 of constructing the first training data set according to the exemplary embodiment of the present application may include the following sub-steps:

[0082] Step S411, data acquisition. As an exemplary implementation, 24 color cards can be photographed in a darkroom experimental environment using a uniform light, and two sets of data are photographed at each preset ISO (International Standards Organization, representing the sensitivity of the image sensor) value: one set of normal exposure and one set of underexposure. Each set of data is photographed for several times and the device must be kept stationary. After completing the data capture of one ISO value, adjust the ISO value and repeat the capture. To ensure the number of available sample points, data at no less than 8 preset ISO values ​​must be captured.

[0083] Step S412, calibrate the noise model of the camera module. According to the data captured in step S411, the mean and variance of each pixel under a single ISO value are counted, and a linear noise model is fitted: y = kx + b, where x represents the brightness mean of the pixel, y represents the noise intensity (ie, variance), and k and b are parameters of the noise model.

[0084] After obtaining the noise intensity parameters at each preset ISO value, the corresponding relationship between the parameters and the ISO value is respectively fitted using a polynomial, so as to obtain the noise intensity at any ISO value and any brightness. The calibration of the noise model can be completed through the above steps S411 and S412.

[0085] Step S421, extracting high-definition image frames according to high-definition video data. The high-definition video data may be, for example, high-definition video data acquired and output by an image sensor, or may be acquired in other ways. For example, high-definition image frames may also be extracted from high-definition image data, as long as the image data or video data meets the high-definition format.

[0086] Step S422: performing degradation processing on the image frames extracted from the video data.

[0087] Since the first image processing model includes a demosaicing function, paired Raw-RGB data pairs are required to construct a training data set. In this embodiment, the Raw-RGB training data pairs can be obtained by performing some processing such as brightness and color degradation on the image frames extracted from the high-definition video.

[0088] As an exemplary embodiment, the degradation process in step S422 may include the following processes: inverse gamma correction, inverse dynamic range compression, inverse color correction and inverse white balance. To simplify the operation, inverse gamma correction with random parameters may be used instead of inverse dynamic range compression to achieve a similar effect.

[0089] Step S423: extract data from the image frame after the degradation processing in step S422 according to the set Bayer format to obtain degraded Raw data.

[0090] Step S424: adding noise to the degraded Raw data according to a pre-calibrated noise model to obtain noisy Raw data.

[0091] Step S425, sharpening the extracted image frame to make the image edge clearer. Step S425 can be performed simultaneously with step S422.

[0092] Step S426, performing the same degradation processing as step S422 on the sharpened image frame to obtain degraded RGB data.

[0093] In step S430, the noisy Raw data obtained in step S424 and the degraded RGB data obtained in step S426 are used as a first training data pair to construct a first training data set, and the first image processing model is trained until the model converges.

[0094] In the above process 400, the three processes of noise reduction, de-mosaicing and sharpening are implemented by adopting the first image processing model, which effectively saves computing resource overhead; in addition, a more accurate noise model is obtained by calibrating the noise model, so the noise reduction effect is significantly better than the traditional noise reduction scheme.

[0095] As an exemplary implementation, the brightness adjustment process implemented by the second image processing model in this embodiment may include dynamic range compression and gamma correction. The construction of the training data set is still the core of constructing the second image processing model. In addition, the color consistency problem needs to be considered when constructing the training data set to ensure that the color is not distorted while adjusting the image brightness. Exemplarily, the second image processing model can be trained based on the constructed second training data set. For example, the second training data pair can be obtained based on the original dark image and the target bright image to construct the second training data set.

[0096] like Figure 5 As shown, the process 500 of constructing the second training data set according to the exemplary embodiment of the present application may include the following sub-steps:

[0097] Step S511, converting the original dark image into HSV (Hue Saturation Value) color space.

[0098] Since the adjustment of the image brightness by the second image processing model is nonlinear, it is difficult to construct the second training data set by synthesis. In this embodiment, the second training data pair can be obtained by using the original dark image and the target bright image with open source light and dark data pairs. Alternatively, the corresponding target bright image can be obtained by taking a real dark image and modifying it with a photo editing tool.

[0099] Step S512, performing inverse gamma correction processing on the V channel of the HSV color space of the original dark image.

[0100] Since the original dark image is an image that has been processed by traditional ISP, in order to make the second image processing model include a gamma correction function, in this embodiment, the original dark image is converted to the HSV color space and the V channel thereof is subjected to inverse gamma correction processing.

[0101] Step S513, converting the target light image corresponding to the original dark image into the HSV color space.

[0102] Step S514: merge the H channel and S channel of the target bright image with the inverse gamma-corrected V channel of the original dark image, and convert them into RGB color space to obtain an updated dark image.

[0103] Since there is often a color difference between the original dark image and the target bright image, in order to enable the second image processing model to learn the brightness enhancement, the color difference between the target bright image and the original dark image needs to be eliminated. Therefore, the H and S channels of the bright image and the V channel of the dark image after inverse gamma correction are taken and re-synthesized to obtain the updated dark image.

[0104] Step S520: Use the updated dark image and the target bright image as a second training data pair, construct a second training data set to train the second image processing model until the model converges.

[0105] In the above process 500, dynamic range compression and gamma correction are implemented by adopting the second image processing network, which saves computing resources and achieves better effects on the overall brightness and local contrast of the image.

[0106] According to the image processing method provided in this embodiment, through in-depth analysis of low-light scene noise, color and other problems, the low-light image enhancement algorithm is combined with the traditional ISP algorithm, and the powerful learning ability of the AI ​​algorithm makes up for the shortcomings of the traditional algorithm in terms of effect, and improves the defects of noise residue, color distortion, brightness and contrast in the traditional solution. The image processing method can enhance low-light images, and can retain more image details including color information, and can improve the overall brightness of the image and the targeted local contrast, thereby greatly improving the imaging quality of low-light images.

[0107] Embodiment 2

[0108] This embodiment provides an image signal processor ISP, such as Figure 6 As shown, the ISP may include:

[0109] The first image processing model 110 is used to perform detail enhancement processing on the low-light raw image data collected by the image sensor to obtain first image data. The detail enhancement processing at least includes de-mosaic processing. Exemplarily, the detail enhancement processing implemented by the first image processing model 110 may include noise reduction, de-mosaic processing and sharpening.

[0110] The image correction module 120 is used to perform image correction processing on the first image data to obtain second image data. The image correction module 120 may further include a black level correction unit, an automatic white balance unit, a lens shading correction unit, and a color correction unit, and perform black level correction, automatic white balance, lens shading correction, and color correction processing on the first image in sequence.

[0111] The second image processing model 130 is used to perform brightness adjustment processing on the second image data to obtain target image data. The brightness adjustment processing at least includes gamma correction. Exemplarily, the brightness adjustment processing implemented by the second image processing model 130 may include dynamic range compression and gamma correction.

[0112] The specific processing of the first image processing model 110, the image correction module 120 and the second image processing model 130 and the technical effects thereof can be referred to in the embodiments of the ISP of this embodiment. Figure 2 The relevant descriptions of steps S1 to S3 in the corresponding implementation manner are not repeated here.

[0113] As an exemplary implementation, the ISP of this embodiment may also include an input module 140 and an output module 150. The input module 140 may be used to receive the original image data collected by the image sensor as data to be processed; the output module 150 may be used to obtain standard RGB image data based on the target image data and output it.

[0114] According to some embodiments of the present application, the first image processing model 110 and the second image processing model 130 may be a deep convolutional neural network (CNN) model or a recurrent neural network (RNN) model.

[0115] Compared with traditional solutions, the image signal processor according to the implementation mode of the present application integrates the image processing algorithm based on deep learning into the ISP architecture, and adopts two neural network models to replace some traditional algorithm modules, thereby simplifying the ISP link and the entire data processing flow; and, by combining the low-illumination image enhancement algorithm with the traditional ISP algorithm, the problem that the image enhancement effect of the traditional solution is limited by the front-end ISP module is solved, thereby improving the enhancement effect of low-illumination scene images.

[0116] In addition, according to an embodiment of the present application, an image processing system is also provided, which may include an image sensor (Sensor) and an image signal processor (ISP) according to the above embodiment of the present application. In some embodiments, the image processing system may also include a memory and a controller, etc.

[0117] In an exemplary embodiment, the Sensor outputs the collected raw data to the ISP, and the ISP uses the raw image data as the data to be processed, and performs detail enhancement processing, image correction processing, and brightness adjustment processing on the data to be processed according to the image processing method in the above embodiment, and then obtains and outputs the target image information. The relevant content of the ISP processing the raw image data can be referred to the description of the above embodiment, and for the sake of brevity, it will not be repeated here.

[0118] In addition, according to an embodiment of the present application, an electronic device 700 is also provided, such as Figure 7 As shown, the electronic device 700 may include: at least one processor and a memory, the memory is communicatively connected to the at least one processor and stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned methods 200, 400 and / or 500, etc.

[0119] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above method 200 is implemented.

[0120] Figure 7 The block diagram of an electronic device 700 suitable for implementing the exemplary embodiment of the present application is schematically shown. The electronic device 700 may be used to implement, for example, Figure 2 , Figure 4 and Figure 5 The apparatus of each method or step described in. Figure 7As shown, the electronic device 700 includes a processor 701, which can perform various appropriate steps and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0121] Multiple components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a touch screen, a handle, etc.; an output unit 707, such as various types of displays, speakers, etc.; a memory 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0122] The processor 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 701 may perform the various methods and processes described above, such as executing methods 200, 400, and / or 500. For example, in some embodiments, methods 200, 400, and / or 500 may be implemented as a computer software program, which is stored in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the processor 701, one or more steps in the method described above may be performed. Optionally, in other embodiments, the processor 701 may be configured to execute one or more steps in the method in any other appropriate manner (eg, by means of firmware).

[0123] It should be further noted that the present application may include methods, devices, systems and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present application.

[0124] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used in this application is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0125] The computer-readable program instructions described in this application can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0126] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as C language or similar programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect to the Internet). In certain embodiments, by using the state information of computer-readable program instructions to customize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present application.

[0127] Various aspects of the present application are described in this application with reference to flowcharts of methods, devices (systems), and computer program products according to exemplary embodiments of the present application. It should be understood that each step of the flowchart and the combination of the steps in the flowchart can be implemented by computer-readable program instructions.

[0128] These computer-readable program instructions can be provided to a processor in a voice interaction device, a general-purpose computer, a special-purpose computer, or a processing unit of other programmable data processing devices, thereby producing a machine, so that when these instructions are executed by the processing unit of a computer or other programmable data processing device, a device that implements the functions / steps specified in one or more steps in the flowchart is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / steps specified in one or more steps in the flowchart.

[0129] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / steps specified in one or more steps in the flowchart.

[0130] The flowchart in the accompanying drawings shows the possible architecture, functions and operations of the equipment, methods and computer program products according to multiple embodiments of the present application. In this regard, each step in the flowchart can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative embodiments, the functions marked in the steps can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each step in the flowchart, and the combination of the steps in the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. An image processing method applied to an image signal processor, comprising: Performing detail enhancement processing on original image data collected by the image sensor based on the first image processing model to obtain first image data; Performing image correction processing on the first image data to obtain second image data; as well as Performing brightness adjustment processing on the second image data based on the second image processing model to obtain target image data, The first image processing model is pre-trained based on preset original data and RGB data, and the second image processing model is pre-trained based on preset original dark image and target bright image data.

2. The method according to claim 1, wherein: The method further comprises: Performing degradation processing on the acquired high-definition image frame to obtain degraded original data; Using a pre-calibrated noise model to superimpose noise on the degraded original data to obtain noisy original data; Performing sharpening processing and degradation processing on the high-definition image frame to obtain degraded RGB data; The noisy original data and the degraded RGB data are used as a first training data pair to construct a first training data set to train the first image processing model.

3. The method according to claim 2, wherein: Performing degradation processing on the high-definition image frame to obtain degraded original data includes: The high-definition image frame is subjected to degradation processing, and data is extracted according to a set Bayer format to obtain degraded original data.

4. The method according to claim 2, wherein: The degradation processing includes the following processing: inverse gamma correction, inverse dynamic range compression, inverse color correction and inverse white balance.

5. The method according to claim 2, wherein: The pre-calibrated noise model is obtained according to the following method: Obtaining data of preset ISO values ​​in a darkroom experimental environment, wherein two sets of data including normal exposure data and underexposure data are respectively captured at each of the preset ISO values; The mean and variance of each pixel at each preset ISO value are statistically calculated according to the captured data to obtain a noise intensity parameter for each preset ISO value, and a linear noise model is fitted as the noise model.

6. The method according to claim 1, wherein: The method further comprises: Convert the original dark image to the HSV color space and perform inverse gamma correction on the brightness V channel of the HSV color space; Convert the target bright image to HSV color space and extract the hue H channel and saturation S channel of the HSV color space; Merging the H channel and the S channel of the target bright image with the V channel of the original dark image that has been subjected to inverse gamma correction, and converting them into an RGB color space to obtain an updated dark image; The updated dark image and the target bright image are used as a second training data pair to construct the second training data set to train the second image processing model.

7. An image signal processor, characterized in that: include: A first image processing model is used to perform detail enhancement processing on the original image data collected by the image sensor to obtain first image data; An image correction module, used for performing image correction processing on the first image data to obtain second image data; as well as A second image processing model is used to perform brightness adjustment processing on the second image data to obtain target image data, The first image processing model is pre-trained based on preset original data and RGB data, and the second image processing model is pre-trained based on preset original dark image and target bright image data.

8. An image processing system, comprising an image sensor for collecting raw image data to be processed, characterized in that: Also includes the image signal processor as claimed in claim 7.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having machine-executable instructions stored thereon, wherein when the machine-executable instructions are executed, the machine is enabled to perform the method according to any one of claims 1 to 6.

Images