Color restoration method and device

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a color restoration method and device. According to the technical scheme, the white balance parameter and the color correction parameter are calculated for the image data collected by the camera under different color temperatures and different brightness, the mapping relation between the color temperature data and the image average brightness and the white balance parameter and the color correction parameter is established, and when image color correction needs to be carried out subsequently, the image color correction efficiency is improved. The image average brightness and the white balance parameter can be quickly determined according to the color temperature data and the image average brightness of the image in combination with the mapping relation, so that color restoration correction is performed on the image, the calculation efficiency is improved while a good restoration effect is achieved, and the method has better applicability for different application scenes.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a color restoration method and device. Background Art

[0002] Usually, the response of the camera sensor to the spectrum in each RGB component deviates from the response of the human eye to the spectrum, and correction needs to be made on the cross effect and each component. The usual practice is to correct the color through a color correction matrix. The purpose of white balance is to perform different gains on each color channel to remove the influence of the color temperature of the light source, and ultimately make the data output by the sensor after being sensitive to a white object under a certain light source display white, solving the problem of green color in the picture.

[0003] At present, the mainstream methods of color correction are divided into two types: traditional ISP processing and deep learning-based methods. Traditional white balance algorithms mainly include two categories. One is the white balance algorithm based on assumptions: mainly gray world method, perfect reflection method and a series of improved algorithms for these two algorithms.

[0004] The gray world method assumes that the average value of the RGB channel pixels in the image is a constant representing the grayscale, and the gain coefficient is obtained by the ratio of the mean value of each channel to the grayscale constant. The algorithm is simple to calculate, but it cannot correct monochrome images.

[0005] The perfect reflection method assumes that the brightest point in the image is the white point, but when there are strong light points in the image, the correction effect is poor. The other type is the white balance algorithm based on prior knowledge. Although the correction effect is better, the calculation amount is large and the applicability is poor.

[0006] In the color correction method based on deep learning, the model based on the end-to-end architecture has a large amount of calculation, which makes it difficult to achieve real-time results on the edge or PC side. On the other hand, it is difficult to obtain high-quality data sets, resulting in poor model generalization ability. Another method based on the CNN framework to output CCM and white balance parameters can solve the problem of large amount of calculation, but it is very difficult to build an accurate data set of input images and parameter pairs. The CCM parameters of each image data need to be optimized with the help of the X-rite24 color card.

[0007] The above method has the following disadvantages:

[0008] 1. The traditional white balance algorithm has poor color restoration effect for large areas of pure color, especially for large areas of skin color scenes, the color is often inaccurate and the color restoration effect in dark environments is poor.

[0009] 2. When there are strong light spots in the image, the correction effect of the perfect reflection method is poor. Although the white balance algorithm based on prior knowledge has a good correction effect, it has a large amount of calculation and poor applicability.

[0010] 3. The end-to-end architecture model based on deep learning has a large amount of computation, and it is difficult to achieve real-time results on the edge or PC side, and the model generalization ability is poor.

[0011] 4. In the microscope environment, it is difficult to correct the color because the X-Rite24 color chart cannot be obtained. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide a color restoration method and device to achieve color restoration correction of images.

[0013] In order to solve the above technical problems, a technical solution adopted by the present invention is:

[0014] A color restoration method comprises the following steps:

[0015] Obtain image data collected by the camera at different color temperatures and brightness, and convert it into RGB images to calculate white balance parameters and color correction parameters;

[0016] Acquire color temperature data and average brightness of the image, and establish a mapping relationship between the color temperature data, the average brightness of the image and the white balance parameter and the color correction parameter;

[0017] The pre-processed target Raw image is processed by the color temperature prediction model to obtain the target color temperature data;

[0018] Calculating the target image average brightness of the target Raw image, and determining the target white balance parameter and the target color correction parameter through the mapping relationship in combination with the target color temperature data;

[0019] The target Raw image is color restored using the target white balance parameter and the target color correction parameter.

[0020] In order to solve the above technical problems, a technical solution adopted by the present invention is:

[0021] A color restoration device, comprising:

[0022] Color temperature and parameter mapping module: used to establish the mapping relationship between color temperature data, image average brightness and white balance parameters, and color correction parameters;

[0023] Dataset construction module: used to construct the training dataset of the color temperature prediction model;

[0024] Model building and training module: used to build the color temperature prediction model and train the color temperature prediction model;

[0025] Model quantification and deployment module: used to quantify and deploy the color temperature prediction model;

[0026] And the steps in the above-mentioned color restoration method are implemented through the above-mentioned modules.

[0027] The beneficial effects of the present invention are:

[0028] The present invention provides a color restoration method and device, which calculates white balance parameters and color correction parameters for image data collected by a camera at different color temperatures and different brightnesses, establishes a mapping relationship between color temperature data, image average brightness, white balance parameters, and color correction parameters, and when image color correction is required later, the image average brightness and white balance parameters can be quickly determined based on the image color temperature data, image average brightness, and the mapping relationship, thereby performing color restoration correction on the image, which has a good restoration effect while improving calculation efficiency and has better applicability for different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A brief flow chart of a color restoration method of the present invention;

[0030] Figure 2 A partial flow chart of a mapping relationship establishment step of a color restoration method of the present invention;

[0031] Figure 3 Constructing a training example graph for a color temperature prediction model of a color restoration method of the present invention;

[0032] Figure 4 A flow chart of the steps of a color restoration method of the present invention;

[0033] Figure 5 is a structural diagram of a color restoration device of the present invention;

[0034] Description of labels:

[0035] 1. A color restoration device; 2. Color temperature and parameter mapping module; 3. Data set construction module; 4. Model building and training module; 5. Model quantization and deployment module. DETAILED DESCRIPTION

[0036] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0037] Please refer to Figure 1 , a color restoration method, comprising the following steps:

[0038] Obtain image data collected by the camera at different color temperatures and brightness, and convert it into RGB images to calculate white balance parameters and color correction parameters;

[0039] Acquire color temperature data and average brightness of the image, and establish a mapping relationship between the color temperature data, the average brightness of the image and the white balance parameter and the color correction parameter;

[0040] The pre-processed target Raw image is processed by the color temperature prediction model to obtain the target color temperature data;

[0041] Calculating the target image average brightness of the target Raw image, and determining the target white balance parameter and the target color correction parameter through the mapping relationship in combination with the target color temperature data;

[0042] The target Raw image is color restored using the target white balance parameter and the target color correction parameter.

[0043] From the above description, it can be seen that the beneficial effects of the present invention are: a color restoration method provided by the present invention calculates white balance parameters and color correction parameters for image data collected by a camera at different color temperatures and different brightnesses, and establishes a mapping relationship between color temperature data, image average brightness and white balance parameters, and color correction parameters. When image color correction is required later, the image average brightness and white balance parameters can be quickly determined based on the image color temperature data, image average brightness and the mapping relationship, thereby performing color restoration correction on the image. While having a good restoration effect, it improves calculation efficiency and has better applicability for different application scenarios.

[0044] Furthermore, obtaining image data collected by the camera at different color temperatures and different brightnesses and converting the image data into an RGB image to calculate white balance parameters and color correction parameters includes the following steps:

[0045] Collect the Raw data of the standard color card under different color temperature environments and brightness, and implement the following steps for each Raw data:

[0046] Extracting a regional image of the standard color card area in the Raw data, segmenting and sorting the extracted regional image into color blocks, and converting the extracted regional image into RGB data;

[0047] Calculate white balance data based on the R, G, and B channel pixel values ​​according to a preset white block area for the RGB data;

[0048] Calculating color correction parameters for the RGB data based on color space conversion and color difference optimization;

[0049] Calculate the average brightness of the image.

[0050] From the above description, it can be seen that the image data of the standard color card under different color temperature environments and brightnesses are collected, and the model white balance data, color correction parameters and image average brightness are calculated, so that the mapping relationship between the color temperature data, image average brightness and white balance parameters and color correction parameters established subsequently can have a more accurate color restoration effect for different colors.

[0051] Furthermore, the calculation of the white balance parameters comprises the steps of:

[0052] Calculate the mean of the R, G, and B channel pixel values ​​of several preset white block areas;

[0053] Calculate the ratio of the G channel pixel value to the R channel pixel value and the B channel pixel value to obtain the gain values ​​of the R channel and the B channel of the white color block;

[0054] According to the average of the gain values ​​of the R channel and the B channel of a plurality of white color blocks, a white balance parameter including a B channel gain and an R channel gain is determined.

[0055] It can be seen from the above description that before establishing the mapping relationship, the white balance parameter calculation needs to convert the image into an RGB image and calculate and determine it based on the pixel gains of the R, G, and B channels.

[0056] Furthermore, the mapping relationship between the image average brightness L, the color temperature data CT and the white balance parameter is determined as follows:

[0057] GainR=f(CT,L);

[0058] minh(GainR,GainR gt );

[0059] GainB=f(CT,L);

[0060] minh(GainB,GainB gt );

[0061] Among them, GainR is the R channel gain calculated by CT and L, GainRgt is the actual R channel gain calculated by the color temperature box environmental data; GainB is the B channel gain calculated by CT and L, GainBgt is the actual B channel gain calculated by the color temperature box environmental data.

[0062] From the above description, it can be seen that based on the known average brightness of the image, color temperature data and white balance parameters, the relationship function between the three is calculated, and the relationship function with the smallest difference between the white balance parameters calculated by the function and the actual white balance parameters is used as the mapping relationship between the average brightness of the image, the color temperature data and the white balance parameters.

[0063] Furthermore, the mapping relationship between the image average brightness L, the color temperature data CT and the color correction parameter CCM is determined as follows:

[0064]

[0065] rr=f(CT,L)minh(rr,rr gt )

[0066] gr=f(CT,L)minh(gr,gr gt )

[0067] br=f(CT,L)minh(br,br gt );

[0068] rg=f(CT,L)minh(rg,rg gt );

[0069] gg=f(CT,L)minh(gg,gg gt );

[0070] bg=f(CT,L)minh(bg,bg gt );

[0071] rb=f(CT,L)minh(rb,rb gt );

[0072] gb=f(CT,L)minh(gb,gb gt );

[0073] bb=f(CT,L)minh(bb,bb gt );

[0074] Among them, rr, gr, br, rg, gg, bg, rb, gb and bb are 9 parameters in the color correction parameter CCM. gt ,gr gt ,br gt , rg gt ,gg gt 、bg gt ,rb gt 、gb gt and bb gt They are respectively values ​​calculated for the 9 parameters in the color correction parameter CCM corresponding to the color temperature box environment data.

[0075] From the above description, it can be seen that, similarly, according to the known image average brightness, color temperature data and various parameter values ​​in the color correction parameters, the relationship function is calculated, and the relationship function with the smallest difference between the function calculation and the actual value is used as the mapping relationship.

[0076] Furthermore, the preprocessed target Raw image is processed by the color temperature prediction model to obtain the target color temperature data, including the following steps:

[0077] For the target Raw image, the correction algorithm is used to eliminate the influence of the inherent non-uniformity of the sensor, and the black level pixel value in the image is adjusted to a reference value, and then the resolution is reduced and converted into RGB data;

[0078] After the RGB data is normalized, it is input into a pre-trained and quantized color temperature prediction model to obtain target color temperature data output by the color temperature prediction model.

[0079] From the above description, it can be seen that for the target image that needs to be color restored, the color temperature data is acquired through the trained color temperature prediction model to solve the problem of unknown image color temperature data and facilitate the subsequent calculation of white balance parameters and color correction parameters.

[0080] Furthermore, the construction of the training data set of the color temperature prediction model includes the steps of:

[0081] Acquire image data of different light source color temperatures, illuminances, image blur levels, and sensor types to obtain a first data set;

[0082] Acquire a data set of standard color card images at different color temperatures and different brightnesses to obtain a second data set;

[0083] For the images in the first data set and the second data set, the images are divided into small image blocks according to the input size of the color temperature prediction model, and the labels of the small image blocks are set to the corresponding color temperature data / 10000;

[0084] Generate a training dataset.

[0085] It can be seen from the above description that, through the above method, a training data set is established for training the color temperature prediction model.

[0086] Furthermore, the training of the color temperature prediction model includes the steps of:

[0087] Acquire a training data set, and preprocess the training data set;

[0088] Setting hyperparameters for the color temperature prediction model, wherein the hyperparameters include a learning rate, a number of samples per iteration, a number of iterations, and a learning rate decay strategy;

[0089] The color temperature prediction model is trained according to the training data.

[0090] From the above description, it can be seen that by setting the hyperparameters of the color temperature prediction model, the color temperature prediction model is continuously optimized during the training process, thereby achieving a more accurate prediction effect.

[0091] Furthermore, the learning rate decay strategy includes:

[0092] In the first preset number of rounds, the learning rate is adjusted upward round by round, and in subsequent training, the learning rate is adjusted downward round by round.

[0093] From the above description, it can be seen that based on the above learning decay strategy, model training can be better implemented in the current scenario.

[0094] Please refer to Figure 2 , a color restoration device, comprising:

[0095] Color temperature and parameter mapping module: used to establish the mapping relationship between color temperature data, image average brightness and white balance parameters, and color correction parameters;

[0096] Dataset construction module: used to construct the training dataset of the color temperature prediction model;

[0097] Model building and training module: used to build the color temperature prediction model and train the color temperature prediction model;

[0098] Model quantification and deployment module: used to quantify and deploy the color temperature prediction model;

[0099] And the steps in the above-mentioned color restoration method are implemented through the above-mentioned modules.

[0100] From the above description, it can be seen that the beneficial effects of the present invention are: a color restoration device provided by the present invention, based on a color temperature and parameter mapping module, a data set construction module, a model building and training module, and a model quantization and deployment module, implements the steps in the above-mentioned color restoration method, and can quickly determine the image average brightness and white balance parameters according to the color temperature data of the image, the image average brightness and the mapping relationship, thereby performing color restoration and correction on the image, while having a good restoration effect, improving the calculation efficiency, and having better applicability for different application scenarios.

[0101] A color restoration method and system of the present invention are suitable for color restoration correction of images captured by different types of image acquisition devices in different display environments. The image acquisition devices include but are not limited to cameras, camcorders and mobile phones of various brands and models. The display environments include but are not limited to conventional environments and even different display device environments. Different display device environments include but are not limited to microscope environments.

[0102] Please refer to 1 and Figure 2, Embodiment 1 of the present invention is:

[0103] A color restoration method is applied to an image acquisition device in this embodiment to perform color restoration correction on images acquired under a normal environment, and the implementation includes the following steps:

[0104] Obtain image data collected by a camera at different color temperatures and brightnesses, and convert the image into an RGB image to calculate white balance parameters and color correction parameters; including the following steps:

[0105] Collect the Raw data of the standard color card under different color temperature environments and brightness, and implement the following steps for each Raw data:

[0106] Extracting a regional image of the standard color card area in the Raw data, segmenting and sorting the extracted regional image into color blocks, and converting the extracted regional image into RGB data;

[0107] Calculate white balance data based on the R, G, and B channel pixel values ​​according to a preset white block area for the RGB data;

[0108] Calculating color correction parameters for the RGB data based on color space conversion and color difference optimization;

[0109] Calculate the average brightness of the image.

[0110] In this embodiment, refer to Figure 2 , camera data acquisition, using the color temperature box environment to collect X-Rite 24 color card background Raw data:

[0111] First, set the camera parameters. When collecting data sets in the Raw format, turn off the camera's ISP function. Then adjust the exposure time, focal length, aperture, and other related parameters. The field of view can be blurred or clear, but at least collect a clear picture to compare with the real card color to determine the accuracy of the white balance parameters and CCM parameters.

[0112] After setting the relevant parameters, ensure that the X-Rite 24 color card is in the field of view and move the camera or target to diversify the target background. The data background brightness is properly adjusted, and each color temperature environment contains multiple brightness. Each model of camera collects multiple color temperatures including 2500-7500K, and a set of data is taken every 100K.

[0113] In this embodiment, the corresponding white balance and color correction parameters CCM parameters are calculated according to the collected data:

[0114] Get the 24 color block areas of the X-Rite 24 color card in the Raw image data.

[0115] First, we use deep learning-based target detection models, including but not limited to Yolo series such as yolov5, yolov8, SSD, R-CNN series, etc. to train the X-Rite 24 color card area detection model. The function of this detection model is to find the location of the X-Rite 24 color card area from the Raw data.

[0116] Segment the X-Rite 24 color card area. Use deep learning-based segmentation models, including but not limited to SAM, Mask RCNN, U2Net and other segmentation models, to segment the 24 color blocks in the 24 color card. The segmented modules remove incorrect color blocks by limiting conditions such as area, aspect ratio, etc.

[0117] If the number of color blocks is not equal to 24, it is necessary to check for missing color blocks. Cluster and sort the color blocks in the x and y directions according to their positions. Each row in the x direction has 6 color blocks, and each row in the y direction has 4 color blocks. If the number is less than a certain number, fill it in according to the overall distribution of the color blocks.

[0118] Sort the 24-color card area. Based on the same R / G and B / G ratios of color blocks 19 to 24, find color blocks 19 to 24 in the 24-color card area, and then rotate the entire color block according to the position and order of color blocks 19 to 24 to ensure that the last row is color blocks 19 to 24.

[0119] The traditional color correction method is generally to manually mark the X-rite24 color card board, and then estimate the position of the 24 color blocks by the color card position. This method is often inefficient and the cut color blocks may contain boundaries. The unclean color block area has a great impact on the subsequent calculation of the white balance gain coefficient and CCM. In the present invention, the X-rite 24 color card board is first detected using a deep learning-based detection model, and then the 24 small color blocks in the X-rite24 color card board are accurately segmented using a segmentation model. The deep learning-based method is not only efficient and fast, but also has good adaptability to lighting, rotation and size. It can also perfectly find the 24 color block areas to ensure the accuracy of subsequent parameter calculations.

[0120] The calculation of the white balance parameters comprises the steps of:

[0121] Calculate the mean of the R, G, and B channel pixel values ​​of several preset white block areas;

[0122] Calculate the ratio of the G channel pixel value to the R channel pixel value and the B channel pixel value to obtain the gain values ​​of the R channel and the B channel of the white color block;

[0123] According to the average of the gain values ​​of the R channel and the B channel of a plurality of white color blocks, a white balance parameter including a B channel gain and an R channel gain is determined.

[0124] In this embodiment, the Raw data is first read for black level processing, and then a demosaicing operation is performed to obtain the BGR data corresponding to the Raw data. The R, G, and B mean values ​​of the 20th, 21st, 22nd, and 23rd color block areas are calculated respectively:

[0125]

[0126] AvgR=SumR / NM

[0127]

[0128] AvgG=SumG / NM

[0129]

[0130] AvgB = SumB / NM;

[0131] In the formula, x and y are the horizontal and vertical coordinates of the image, N and M are the width and height of the image, fr(x, y) is the pixel value of the image R channel (x, y) position, fg(x, y) is the pixel value of the image G channel (x, y) position, and fb(x, y) is the pixel value of the image B channel (x, y) position. SumR, SumG, and SumB are the sum of the pixel values ​​of the R, G, and B channels of the color block area, respectively. AvgR, AvgG, and AvgB are the average values ​​of the pixel values ​​of the R, G, and B channels of the color block area, respectively.

[0132] Calculate the ratio of R / G to B / G:

[0133] GainR k =AvgG k / AvgR k

[0134] GainB k =AvgG k / AvgB k ;

[0135] In the formula, GainRk and GainBk are the gain values ​​of the R channel and B channel of the kth color block respectively.

[0136] Finally, calculate the R channel gain GainR and B channel gain GainB in the white balance parameters:

[0137] GainR=(GainR 20 +GainR 21 +GainR 22 +GainR23 ) / 4

[0138] GainB=(GainB 20 +GainB 21 +GainB 22 +GainB 23 ) / 4.

[0139] Color correction parameter CCM calculation:

[0140] The RAW image is processed to obtain the sRGB value, and then the sRGB is converted to xyz and then to Lab color space. The specific conversion formula can be found in CIELAB. The following formula is used to optimize the CCM matrix through the color difference ΔE:

[0141] ΔL=L measured -L ideal

[0142] Δa=a measured -a ideal

[0143] Δb=b measured -b ideal

[0144]

[0145] Where ΔE is the color difference, measured is the measured value, and ideal is the theoretical value, also known as the X-Rite nominal value.

[0146] The calculation of the average brightness of the image is as follows:

[0147] The average brightness of the image is calculated using the weighted average method based on the R channel pixel value, G channel pixel value, and B channel pixel value of all pixels in the RGB image. The calculation formula can be referred to in the following example:

[0148]

[0149] Where w is the image width, h is the image height, f(i, j)R is the R channel pixel value at image (i, j), f(i, j)G is the G channel pixel value at image (i, j), and f(i, j)B is the B channel pixel value at image (i, j).

[0150] Acquire color temperature data and average brightness of the image, and establish a mapping relationship between the color temperature data, the average brightness of the image and the white balance parameter and the color correction parameter;

[0151] Determination of the mapping relationship between the image average brightness L, the color temperature data CT and the white balance parameter:

[0152] GainR=f(CT,L);

[0153] minh(GainR,GainR gt );

[0154] GainB=f(CT,L);

[0155] minh(GainB,GainB gt );

[0156] Among them, GainR is the R channel gain calculated by CT and L, GainRgt is the actual R channel gain calculated by the color temperature box environmental data; GainB is the B channel gain calculated by CT and L, GainBgt is the actual B channel gain calculated by the color temperature box environmental data.

[0157] In this embodiment, the white balance parameters GainR and GainB are calculated by the color temperature (CT) and the average brightness of the image. Since the white balance parameters are different at different color temperatures and the white balance parameters are also different at different brightnesses at the same color temperature, it is necessary to calculate the white balance parameters at different color temperatures and brightnesses respectively.

[0158] Determination of the mapping relationship between the image average brightness L, the color temperature data CT and the color correction parameter CCM:

[0159]

[0160] rr=f(CT,L)minh(rr,rr gt )

[0161] gr=f(CT,L)minh(gr,gr gt )

[0162] br=f(CT,L)minh(br,br gt );

[0163] rg=f(CT,L)minh(rg,rg gt );

[0164] gg=f(CT,L)minh(gg,gg gt );

[0165] bg=f(CT,L)minh(bg,bg gt );

[0166] rb=f(CT,L)minh(rb,rb gt );

[0167] gb=f(CT,L)minh(gb,gb gt );

[0168] bb=f(CT,L)minh(bb,bb gt );

[0169] Among them, rr, gr, br, rg, gg, bg, rb, gb and bb are 9 parameters in the color correction parameter CCM. gt ,gr gt ,br gt , rg gt ,gg gt 、bg gt ,rb gt 、gb gt and bb gt They are respectively values ​​calculated for the 9 parameters in the color correction parameter CCM corresponding to the color temperature box environment data.

[0170] In this embodiment, nine parameters in CCM are calculated respectively by using the color temperature CT and the image average brightness L.

[0171] The pre-processed target Raw image is processed by the color temperature prediction model to obtain the target color temperature data;

[0172] The target image average brightness of the target Raw image is calculated, and combined with the target color temperature data, a target white balance parameter and a target color correction parameter are determined through the mapping relationship.

[0173] In this embodiment, before inputting the color temperature prediction model, the target Raw image is further subjected to the following steps:

[0174] The target Raw image is processed with black level, downscaled and demosaiced in turn to obtain RGB data.

[0175] After the RGB data is standardized or normalized, it is input into a pre-trained and quantized color temperature prediction model to obtain color temperature data output by the color temperature prediction model.

[0176] And calculate the average brightness of the image based on the RGB data.

[0177] Then, based on the color temperature data, the average brightness of the image and the mapping relationship determined above, the target white balance parameters and the target color correction parameters are calculated.

[0178] The target Raw image is color restored using the target white balance parameter and the target color correction parameter.

[0179] Please refer to Figure 3 , Embodiment 2 of the present invention is:

[0180] A color restoration method is different from the first embodiment in that the training and use of the color temperature prediction model are specifically described in this embodiment.

[0181] In this embodiment, the construction of the training data set of the color temperature prediction model includes the following steps:

[0182] Image data of different light source color temperatures, illuminances, image blur levels, and sensor types are obtained to obtain a first data set.

[0183] In this embodiment, various data in the image acquisition device environment are obtained by controlling multiple variables such as light source color temperature, illumination, image blur degree, and sensor type.

[0184] Acquire a data set of standard color card images at different color temperatures and different brightnesses to obtain a second data set;

[0185] For the images in the first data set and the second data set, the images are divided into small image blocks according to the input size of the color temperature prediction model, and the labels of the small image blocks are set to the corresponding color temperature data / 10000;

[0186] Generate a training dataset.

[0187] In this embodiment, the color temperature box X-Rite 24 color card plate image data set is generated: the RGB image decoded from the Raw data is cut into multiple small blocks according to the model input size based on the calculated white balance parameters and the image brightness mean. The output parameter corresponding to the small block is the corresponding color temperature divided by 10000, so that the color temperature value falls uniformly between 0.0-1.0.

[0188] Similarly, the image data collected by the image acquisition device is cut into multiple small blocks according to the model input size to enrich the diversity of the training data set.

[0189] The training of the color temperature prediction model includes the steps of:

[0190] Acquire a training data set, and preprocess the training data set;

[0191] In this embodiment, the preprocessing of the training data includes but is not limited to black level processing (Black Level) and standardization processing or normalization processing.

[0192] Setting hyperparameters for the color temperature prediction model, wherein the hyperparameters include a learning rate, a number of samples per iteration, a number of iterations, and a learning rate decay strategy;

[0193] The color temperature prediction model is trained according to the training data.

[0194] In this embodiment, a color temperature prediction model is constructed based on CNN. First, the model input is determined to be 256×256×3 or 128×128×3 or other sizes suitable for model calculation, and the model output is the color temperature CT. The loss function of the model can use regression loss functions such as mean square error loss function (L2 Loss), absolute value loss function (L1Loss) and Smooth L1 Loss loss function. The model framework can be the current mainstream model architecture such as: VGGNet, ResNet, InceptionNet, MobileNet, EfficientNet, DenseNet, ShuffleNet, SqueezeNet, RegNet, etc., as well as improved versions based on these model architectures. The last two layers of the model use a fully convolutional neural network.

[0195] Model training includes data reading and preprocessing, hyperparameter setting, model training and model saving. Model training is to obtain the global optimal solution to facilitate subsequent direct parameter calls.

[0196] First, the learning rate change strategy is designed. In this embodiment, the learning rate change adopts a combination of multiple strategies for training. In the first n rounds of training, the learning rate gradually increases, and the subsequent learning rate gradually decreases. The specific learning rate can be dynamically adjusted according to your own data set. The adjustment basis is the decline rate and value of the model loss. The appropriate parameters can be found through multiple adjustments. The purpose of the adjustment is to find the global optimal solution.

[0197] Read the data of the training data set and preprocess the data of the training data set. Read the image data and then perform data enhancement, including image, normalization or standardization operations, data type conversion and other operations.

[0198] The model is subjected to hyperparameter setting, wherein the hyperparameters include a learning rate, the number of samples per iteration, the number of iterations, and a learning rate decay strategy.

[0199] The model is trained using the training set. After setting the parameters, the model is trained and the results are saved. Models based on the Pytorch architecture are saved as .pth or .pt files. Models based on the Tensorflow architecture are saved as .pb files.

[0200] Model testing and verification: Test and verify the obtained model file to ensure that the indicator value corresponding to the model reaches the specified range.

[0201] In this embodiment, the quantitative deployment of the color temperature prediction model includes the following steps:

[0202] For the trained and tested models, Pytorch or Tensorflow tools are used to perform quantization operations on them, converting the model parameter type from double precision to half precision or unsigned integer data type with almost no loss of accuracy, thereby greatly reducing the amount of calculation and enabling them to be deployed on the edge to achieve real-time operation.

[0203] The model takes only 1ms to infer in the GPU environment of PC (CPU: i7-12700, GPU: RTX3080), and only 5ms to infer in the CPU environment of PC (CPU: i7-12700). The FPS of the smart camera based on Xilinx-4EV chip (KV260, B4096 architecture) is close to 100 frames, which fully meets the real-time requirements.

[0204] The quantized model is deployed in the application environment, and the model output is the color temperature data corresponding to the current Raw image.

[0205] In a specific application, the color temperature data output by the image average brightness and color temperature prediction model is referred to the steps in Example 1, and the corresponding white balance parameters and CMM parameters are calculated by the mapping relationship between the color temperature data, brightness and white balance parameters, and CMM parameters to perform color restoration on the collected Raw.

[0206] Please refer to Figure 4 , Embodiment 3 of the present invention is:

[0207] A color restoration method, in this embodiment, the process of color restoration correction of an image captured by an image acquisition device under a microscope environment is described, which can be referred to Figure 4 , which also implements a color restoration method described in the above embodiment one or two.

[0208] Specifically, compared with the color restoration method described in Embodiment 1 or 2, the construction of the training data set of the color temperature prediction model includes the following steps:

[0209] Image data of different light source color temperatures, light intensities, slide types, slide contents, image blur levels, and sensor types are acquired to obtain a first data set.

[0210] In this embodiment, various data in the microscope environment are obtained by controlling multiple variables such as light source color temperature, illumination, slide type, slide content, image blur degree, and Sensor type.

[0211] That is, in other equivalent embodiments, according to different display environments, when conducting model training, by targeted design of the data in the training data set, better color temperature prediction effect can be achieved, more accurate color temperature data can be obtained, and more accurate white balance parameters and color correction parameters can be calculated, thereby enabling more accurate color restoration correction.

[0212] Acquire a data set of standard color card images at different color temperatures and different brightnesses to obtain a second data set;

[0213] The images in the first data set and the second data set are divided into small image blocks according to the input size of the color temperature prediction model, and the labels of the small image blocks are set to the corresponding color temperature data / 10000.

[0214] In this embodiment, the color temperature box X-Rite 24 color card plate image and the microscope environment image need to be cut into multiple small blocks according to the model input size.

[0215] Generate a training dataset.

[0216] Please refer to Figure 5 , Embodiment 4 of the present invention is:

[0217] A color restoration device 1, comprising:

[0218] Color temperature and parameter mapping module 2: used to establish the mapping relationship between color temperature data, image average brightness and white balance parameters, and color correction parameters;

[0219] Dataset construction module 3: training dataset for building color temperature prediction model;

[0220] Model building and training module 4: used to build the color temperature prediction model and train the color temperature prediction model;

[0221] Model quantification and deployment module 5: used to quantify and deploy the color temperature prediction model;

[0222] And the steps in a color restoration method described in any one of the above embodiments 1 to 3 are implemented through the above modules.

[0223] In summary, the color restoration method and device provided by the present invention solve the problems that the traditional white balance algorithm in the microscope environment has poor color restoration effect for large areas of pure color, the deep learning model has a large amount of calculation and poor applicability. In order to solve the above technical problems, the technical solution adopted by the present invention includes a data acquisition module, a data set construction module, a model training module and a model quantization and deployment module. The function of the data acquisition module is to collect Raw data after the camera turns off the ISP, which is used to construct the data pair for model input. The first function of the data set construction module is to measure the white balance and CCM parameters through the X-Rite 24 color card. The second function is to calculate the white balance parameters in the microscope environment by selecting the white point ROI area. The model training module is used to build a model architecture and train it according to the hyperparameters set according to the training set. The model quantization and deployment module is used to convert the data type of the trained model parameters under the premise of ensuring accuracy, saving computing costs, and deploying the quantized model on the edge device side or PC side.

[0224] The microscope environment cannot be color corrected by color card board due to the characteristics of its photography method. In the present invention, the color temperature parameter is used as an intermediate variable to associate the CMM parameters calculated by the color card board with the microscope Raw data, which perfectly solves the problem of model input and output data pairs and microscope environment color correction.

[0225] The model input only requires a small area of ​​the Raw image (256×256×3 or 128×128×3). The model has a small amount of computation and can be deployed in a PC environment, edge device, or embedded device.

[0226] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A color restoration method, characterized in that: The following steps are involved: Obtain image data collected by the camera at different color temperatures and brightness, and convert it into RGB images to calculate white balance parameters and color correction parameters; Acquire color temperature data and average brightness of the image, and establish a mapping relationship between the color temperature data, the average brightness of the image and the white balance parameter and the color correction parameter; The pre-processed target Raw image is processed by the color temperature prediction model to obtain the target color temperature data; Calculating the target image average brightness of the target Raw image, and determining the target white balance parameter and the target color correction parameter through the mapping relationship in combination with the target color temperature data; The target Raw image is color restored using the target white balance parameter and the target color correction parameter.

2. A color restoration method according to claim 1, characterized in that: The steps of obtaining image data collected by the camera at different color temperatures and brightness and converting it into RGB images for calculating white balance parameters and color correction parameters include: Collect the Raw data of the standard color card under different color temperature environments and brightness, and implement the following steps for each Raw data: Extracting a regional image of the standard color card area in the Raw data, segmenting and sorting the extracted regional image into color blocks, and converting the extracted regional image into RGB data; Calculate white balance data based on the R, G, and B channel pixel values ​​according to a preset white block area for the RGB data; Calculating color correction parameters for the RGB data based on color space conversion and color difference optimization; Calculate the average brightness of the image.

3. A color restoration method according to claim 1 or 2, characterized in that: The calculation of the white balance parameters comprises the steps of: Calculate the mean of the R, G, and B channel pixel values ​​of several preset white block areas; Calculate the ratio of the G channel pixel value to the R channel pixel value and the B channel pixel value to obtain the gain values ​​of the R channel and the B channel of the white color block; According to the average of the gain values ​​of the R channel and the B channel of a plurality of white color blocks, a white balance parameter including a B channel gain and an R channel gain is determined.

4. A color restoration method according to claim 1, characterized in that: Determination of the mapping relationship between the image average brightness L, the color temperature data CT and the white balance parameter: GainR=f(CT,L); minh(GainR,GainR gt ); GainB=f(CT,L); minh(GainB,GainB gt ); Among them, GainR is the R channel gain calculated by CT and L, GainRgt is the actual R channel gain calculated by the color temperature box environmental data; GainB is the B channel gain calculated by CT and L, GainBgt is the actual B channel gain calculated by the color temperature box environmental data.

5. A color restoration method according to claim 1, characterized in that: Determination of the mapping relationship between the image average brightness L, the color temperature data CT and the color correction parameter CCM: rr=f(CT,L)minh(rr,rr gt ) gr=f(CT,L)minh(gr,gr gt ) br=f(CT,L)minh(br,br gt ); rg=f(CT,L)minh(rg,rg gt ); gg=f(CT,L)minh(gg,gg gt ); bg=f(CT,L)minh(bg,bg gt ); rb=f(CT,L)minh(rb,rb gt ); gb=f(CT,L)minh(gb,gb gt ); bb=f(CT,L)minh(bb,bb gt ); Among them, rr, gr, br, rg, gg, bg, rb, gb and bb are 9 parameters in the color correction parameter CCM. gt ,gr gt 、br gt , rg gt ,gg gt 、bg gt ,rb gt 、gb gt and bb gt They are respectively values ​​calculated for the 9 parameters in the color correction parameter CCM corresponding to the color temperature box environment data.

6. A color restoration method according to claim 1, characterized in that: Processing the preprocessed target Raw image through the color temperature prediction model to obtain the target color temperature data includes the following steps: For the target Raw image, the correction algorithm is used to eliminate the influence of the inherent non-uniformity of the sensor, and the black level pixel value in the image is adjusted to a reference value, and then the resolution is reduced and converted into RGB data; After the RGB data is normalized, it is input into a pre-trained and quantized color temperature prediction model to obtain target color temperature data output by the color temperature prediction model.

7. A color restoration method according to claim 1, characterized in that: The construction of the training data set of the color temperature prediction model includes the following steps: Acquire image data of different light source color temperatures, illuminances, image blur levels, and sensor types to obtain a first data set; Acquire a data set of standard color card images at different color temperatures and different brightnesses to obtain a second data set; For the images in the first data set and the second data set, the images are divided into small image blocks according to the input size of the color temperature prediction model, and the labels of the small image blocks are set to the corresponding color temperature data / 10000; Generate a training dataset.

8. A color restoration method according to claim 1, characterized in that: The training of the color temperature prediction model includes the steps of: Acquire a training data set, and preprocess the training data set; Setting hyperparameters for the color temperature prediction model, wherein the hyperparameters include a learning rate, a number of samples per iteration, a number of iterations, and a learning rate decay strategy; The color temperature prediction model is trained according to the training data.

9. A color restoration method according to claim 8, characterized in that: The learning rate decay strategy includes: In the first preset number of rounds, the learning rate is adjusted upward round by round, and in subsequent training, the learning rate is adjusted downward round by round.

10. A color restoration device, characterized in that: include: Color temperature and parameter mapping module: used to establish the mapping relationship between color temperature data, image average brightness and white balance parameters, and color correction parameters; Dataset construction module: used to construct the training dataset of the color temperature prediction model; Model building and training module: used to build the color temperature prediction model and train the color temperature prediction model; Model quantification and deployment module: used to quantify and deploy the color temperature prediction model; And the steps in the color restoration method described in any one of claims 1 to 9 are implemented through the above modules.

Images