Cross-sensor white balance data conversion method, integration method, device, equipment, medium and product

Abstract

The invention discloses a cross-sensor white balance data conversion method and device, an integration method and device, equipment, a medium and a product, and relates to the technical field of image processing, and the method comprises the steps: collecting color chart images of a source domain sensor and a target domain sensor under various illumination conditions; for neutral pixels in the color chart, illumination mapping is carried out through a first conversion model, and logarithmic domain data of the neutral pixels are converted from a source domain sensor to a target domain sensor; and for non-neutral pixels in the color chart, logarithmic domain data of the non-neutral pixels are converted from the source domain sensor to the target domain sensor through a second conversion model. The cross-sensor white balance data migration efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and particularly to a cross-sensor white balance data conversion method, an integration method, a cross-sensor white balance data conversion device, a cross-sensor white balance data conversion equipment, a storage medium, and a computer program product. Background Art

[0002] In the field of image processing, the white balance technology is a core link to ensure the color consistency of images under different lighting conditions. Due to the significant differences in the spectral response characteristics of different sensors (such as cameras, mobile phone cameras, etc.), obvious color differences will appear in the images collected by different sensors for the same scene. If the white balance data adapted to the source domain sensor is directly applied to the target domain sensor, it will lead to unsatisfactory results such as color distortion and color temperature deviation corresponding to the target domain sensor.

[0003] To eliminate the unsatisfactory results, the existing technologies usually optimize the white balance data applied to the source domain sensor for the target domain sensor separately, which requires re-collecting a large amount of labeled data, consuming time and effort and being difficult to quickly adapt to the target domain sensor, seriously restricting the migration efficiency of the white balance data. Summary of the Invention

[0004] The main purpose of the present application is to provide a cross-sensor white balance data conversion method, an integration method, a cross-sensor white balance data conversion device, a cross-sensor white balance data conversion equipment, a storage medium, and a computer program product, aiming to solve the technical problem of low efficiency of cross-sensor white balance data migration in the existing technology.

[0005] To achieve the above purpose, the present application proposes a cross-sensor white balance data conversion method, and the method includes:

[0006] Collect color chart images of the source domain sensor and the target domain sensor under various lighting conditions;

[0007] For the neutral pixels in the color chart image, perform illumination mapping through a first conversion model to convert the logarithmic domain data of the neutral pixels from the source domain sensor to the target domain sensor;

[0008] For the non-neutral pixels in the color chart image, convert the logarithmic domain data of the non-neutral pixels from the source domain sensor to the target domain sensor through a second conversion model.

[0009] In an embodiment, before the step of performing illumination mapping through the first conversion model to convert the logarithmic domain data of the neutral pixels from the source domain sensor to the target domain sensor, it includes:

[0010] Under various lighting conditions, the same standard color card is imaged using the source domain sensor and the target domain sensor respectively to obtain multiple pairs of RAW image data;

[0011] From the RAW image data, the response values of the source domain sensor and the target domain sensor to each color block on the color card are extracted to obtain the color feature matrices of the source domain sensor and the target domain sensor in the logarithmic domain;

[0012] The first conversion model is trained through the color feature matrix.

[0013] In one embodiment, the method further includes:

[0014] The color feature matrix of the source domain sensor in the logarithmic domain is feature-expanded to obtain the source domain color feature matrix after feature expansion;

[0015] The first conversion model is trained through the source domain color feature matrix and the color feature matrix of the target domain sensor in the logarithmic domain.

[0016] In one embodiment, the step of performing illumination mapping through the first conversion model to convert the logarithmic domain data of neutral pixels from the source domain sensor to the target domain sensor includes:

[0017] The logarithmic domain data of the neutral pixels of the source domain sensor is input into the first conversion model for prediction of illumination mapping, and the predicted logarithmic domain data of the neutral pixels is output;

[0018] The predicted logarithmic domain data of the neutral pixels is used as the logarithmic domain data of the neutral pixels of the target domain sensor.

[0019] In one embodiment, before the step of converting the logarithmic domain data of non-neutral pixels from the source domain sensor to the target domain sensor through the second conversion model, it includes:

[0020] The illumination distributions corresponding to the neutral pixels of the source domain sensor under various illuminations are fitted to obtain a Planck curve representing the color temperature distribution of the source domain sensor;

[0021] The illumination distributions corresponding to the non-neutral pixels of the source domain sensor under various illuminations are mapped to the Planck curve to obtain the foot points of the logarithmic domain coordinate points of the non-neutral pixels in the coordinate system where the Planck curve is located on the Planck curve;

[0022] The second conversion model is obtained according to the conversion matrix of the foot point and the pre-sampled points adjacent to the foot point.

[0023] In addition, to achieve the above object, the present application also proposes an integration method, and the integration method includes:

[0024] In the data loading stage of training the image processing model of the target domain sensor, the cross-sensor white balance data conversion method is applied to perform cross-sensor white balance data conversion in the data loading stage.

[0025] In addition, to achieve the above object, the present application also proposes a cross-sensor white balance data conversion device, which includes:

[0026] An acquisition module, configured to acquire color chart images of the source domain sensor and the target domain sensor under various illumination conditions;

[0027] A first conversion module, configured to perform illumination mapping on the neutral pixels in the color chart image through a first conversion model, and convert the logarithmic domain data of the neutral pixels from the source domain sensor to the target domain sensor;

[0028] A second conversion module, configured to convert the logarithmic domain data of the non-neutral pixels from the source domain sensor to the target domain sensor through a second conversion model for the non-neutral pixels in the color chart image.

[0029] In addition, to achieve the above object, the present application also proposes a cross-sensor white balance data conversion device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the computer program is configured to implement the cross-sensor white balance data conversion method as described above, and / or the steps of the integrated method.

[0030] In addition, to achieve the above object, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the cross-sensor white balance data conversion method as described above, and / or the steps of the integrated method.

[0031] In addition, to achieve the above object, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the cross-sensor white balance data conversion method as described above, and / or the steps of the integrated method.

[0032] One or more technical solutions proposed by the present application have at least the following technical effects:

[0033] When migrating white balance data across sensors compared with the prior art, it is necessary to re-collect and label data for the target domain sensor. This process not only consumes a large amount of time and labor costs, but also due to the spectral response characteristic differences of different sensors, it is difficult for the white balance data re-optimized on the target domain sensor to be quickly adapted to the target domain sensor, resulting in low efficiency of migrating the white balance data to the target domain sensor. After collecting the color chart images of the source domain sensor and the target domain sensor under various lighting conditions in this application, for the neutral pixels in the color chart images, light mapping is performed through the first conversion model to convert the logarithmic domain data of the neutral pixels from the source domain sensor to the target domain sensor. For the non-neutral pixels in the color chart images, the second conversion model is used to convert the logarithmic domain data of the non-neutral pixels from the source domain sensor to the target domain sensor. This method does not require re-collecting a large amount of labeled data, can effectively decouple the spectral response differences between different sensors, enables the logarithmic domain data of the neutral pixels and the non-neutral pixels of the source domain sensor color chart images to be used as labeled data, and after the labeled data is converted, it can directly adapt to the characteristics of the target domain sensor. In addition to exempting additional data labeling for the target domain sensor, it improves the efficiency of white balance data migration across sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0036] Figure 1 It is a schematic flowchart provided for an embodiment of the cross-sensor white balance data conversion method of the present application;

[0037] Figure 2 It is a schematic flowchart provided for another embodiment of the cross-sensor white balance data conversion method of the present application;

[0038] Figure 3 It is a schematic brief flowchart provided for still another embodiment of the cross-sensor white balance data conversion method of the present application;

[0039] Figure 4 It is a schematic module structure diagram of the cross-sensor white balance data conversion device of the present application;

[0040] Figure 5 It is a schematic device structure diagram of the hardware operating environment involved in the cross-sensor white balance data conversion method of the present application.

[0041] The realization of the purpose, functional features and advantages of this application will be further described in conjunction with embodiments with reference to the accompanying drawings. Specific embodiments

[0042] It should be understood that the specific embodiments described herein are used to explain the technical solutions of this application and are not used to limit this application.

[0043] In order to better understand the technical solutions of this application, the following will be described in detail in conjunction with the drawings of the specification and specific embodiments.

[0044] There is already an A camera (source domain sensor) and white balance data for white balance correction using the white balance algorithm of the A camera; directly applying the white balance algorithm of the A camera to the B camera (target domain sensor) is equivalent to passing the white balance data for white balance correction of the A camera through relevant conversion to conform to the data distribution law of the B camera, so that the B camera can use the white balance algorithm of the A camera for white balance correction.

[0045] This application provides a cross-sensor white balance data conversion method, which converts the white balance data for white balance correction of the source domain sensor and then adapts it to the target domain sensor, and then uses the converted white balance data to train an image processing model for the target domain sensor, avoiding the resource consumption of repeated data annotation in the traditional method and improving the efficiency of white balance data during cross-sensor migration.

[0046] It should be noted that the execution subject of this embodiment can be a cross-sensor white balance data conversion device (hereinafter simply referred to as the data conversion device), or a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a processor, etc. that can implement the above functions. The following takes the data conversion device as an example to illustrate this embodiment and the following embodiments.

[0047] Based on this, the embodiment of this application provides a cross-sensor white balance data conversion method, with reference to Figure 1 , Figure 1 is a schematic flowchart of an embodiment of the cross-sensor white balance data conversion method of this application.

[0048] In this embodiment, the cross-sensor white balance data conversion method includes steps S10 to S40:

[0049] Step S10, collect color card images of the source domain sensor and the target domain sensor under various lighting conditions;

[0050] It should be noted that the source domain sensor and the target domain sensor are two image acquisition devices for white balance data conversion, such as cameras, mobile phone cameras, etc.; the color chart is a RAW format image of standard color patches captured by the sensor. Exemplarily, a color chart with 24 standard color patches can be used, such as Figure 2 As shown, the color values of the color patches are calibrated through standardization and are used to quantify the color response differences between sensors. Multiple lighting conditions refer to acquisitions under H light, A light, U35 light, TL84 light, CWF light, D50 light, D65 light, D75 light, and extended light sources such as LEDs, which are used to simulate the spectral diversity of the actual scene.

[0051] Step S20: For the neutral pixels in the color chart, perform illumination mapping through the first conversion model to convert the logarithmic domain data of the neutral pixels from the source domain sensor to the target domain sensor;

[0052] It should be noted that neutral pixels refer to color patches in the color chart with neutral gray characteristics in spectral reflectance. Exemplarily, in the standard 24-color chart, the neutral pixels refer to the 21st color patch, which does not have a color tendency with the change of illumination color temperature; since there may be differences in the perception of neutral pixels between the source domain sensor and the target domain sensor, it is necessary to eliminate them.

[0053] Implement cross-sensor illumination mapping through the first conversion model to convert the data of neutral pixels in the logarithmic domain from the source domain sensor to the target domain sensor, so as to eliminate the inherent deviation in the neutral gray response between sensors, ensure that the data of neutral pixels in the logarithmic domain can be directly migrated to the target domain sensor, and thus avoid the color deviation problem in the neutral area caused by sensor differences.

[0054] In one embodiment, the conversion method of the logarithmic domain data of neutral pixels can be converted through the second conversion model in addition to the first conversion model.

[0055] In one implementation manner, before step S20, it includes steps A10 to A30:

[0056] Step A10: Under multiple lighting conditions, use the source domain sensor and the target domain sensor respectively to perform image acquisition on the same standard color chart to obtain multiple pairs of RAW image data;

[0057] It should be noted that the acquisition conditions are as follows: in a standardized lighting environment covering different color temperature ranges (such as D50, D65, A light source, etc.), with the lighting, angle, and position kept unified, the standard color card should be placed as horizontally as possible and centered, occupying two-thirds of the screen at the same time, minimizing the influence of shadows and distortions on sampling. At the same time, in order to make the sampling more in line with the distribution of the entire dataset, the color temperatures collected should be as widely distributed and evenly different as possible. Under the above acquisition conditions, the source domain sensor (the original device of the algorithm to be migrated) and the target domain sensor (the new device to be adapted) are used synchronously to collect images of the same standard 24-color card, obtaining multiple pairs of RAW image data with strictly matched light source conditions.

[0058] The paired RAW image data are the original image data captured synchronously by the source domain sensor and the target domain sensor under the same lighting, without compression and color processing. It contains the linear response values of the sensor to each color block of the color card, providing a reference dataset with controllable variables for cross-sensor color conversion.

[0059] Step A20: Extract the response values of the source domain sensor and the target domain sensor to each color block in the color card from the RAW image data, obtaining the color feature matrices of the source domain sensor and the target domain sensor in the logarithmic domain.

[0060] It should be noted that from the RAW image data collected by the source domain sensor and the target domain sensor, locate each color block of the standard color card, calculate the pixel mean of each color block in the RGB channels of the sensor respectively, obtaining the original response values of the color blocks on the source domain sensor and the target domain sensor; perform logarithmic transformation on the original response values to generate logarithmic domain response values. Finally, arrange the logarithmic domain response values of all color blocks in the source domain sensor and the target domain sensor according to the color block numbers, obtaining the color feature matrices of the source domain sensor and the target domain sensor in the logarithmic domain. Among them, each element in the color feature matrix represents the color response value of a specific color block in the logarithmic domain of the sensor (source domain sensor and target domain sensor).

[0061] Exemplarily, if the color card includes 24 color blocks, by averaging the square sub-pixel blocks of each color block, the R, G, and B values of the 24 color blocks are obtained respectively, and then logarithmic transformation is performed through log(G / R) and log(G / B) to obtain a paired color matrix with a shape of [24, 2] in the logarithmic domain, where 24 corresponds to the number of color blocks, and 2 corresponds to the response values of the source domain sensor and the target domain sensor to the color blocks in the logarithmic domain.

[0062] Step A30: Train the first conversion model through the color feature matrix.

[0063] It should be noted that a first conversion model is trained through a machine learning method based on the color feature matrices of the source domain sensor and the target domain sensor. Among them, the machine learning method includes linear regression method, neural network, support vector machine (SVM), decision tree and other machine learning methods. Other machine learning methods can also be selected according to the requirements of specific problems and the characteristics of data to achieve better prediction effects.

[0064] Specifically, in the training process, the logarithmic domain data of the neutral pixels of the source domain sensor is used as the input feature, and the logarithmic domain data of the neutral pixels of the target domain sensor is used as the output feature. The mapping relationship between the input feature and the output feature is fitted through a linear regression algorithm, and finally a first conversion model is formed to establish the illumination mapping rule of the source domain sensor and the target domain sensor on the neutral pixels, and eliminate the response deviation caused by sensor differences.

[0065] In this embodiment, by synchronously collecting the color card RAW image data of the source domain sensor and the target domain sensor under multiple illumination conditions, extracting the color feature matrix in the logarithmic domain, and then training a first conversion model based on the color feature matrix, an illumination-independent linear relationship of the neutral gray response between the source domain sensor and the target domain sensor is established, so that the training of the first conversion model can fully capture the response difference law between sensors and eliminate the response deviation caused by sensor differences. In this embodiment, the response deviation caused by sensor differences is eliminated through the first conversion model, so that the labeled data applied to the source domain sensor can be directly applied to the target domain sensor, realizing cross-sensor data migration, solving the pain point that the existing technology needs to repeatedly re-label the target domain sensor, and improving the efficiency of cross-sensor white balance algorithm deployment.

[0066] In another embodiment, the cross-sensor white balance data conversion method includes:

[0067] Expand the features of the color feature matrix of the source domain sensor in the logarithmic domain to obtain an expanded source domain color feature matrix;

[0068] Train a first conversion model through the source domain color feature matrix and the color feature matrix of the target domain sensor in the logarithmic domain.

[0069] It should be noted that polynomial feature expansion is performed on the color feature matrix of the source domain sensor in the logarithmic domain to generate the source domain color feature matrix in the high-dimensional feature space. Exemplarily, the way of feature expansion can be polynomial feature expansion, introducing non-linear combinations such as quadratic terms and cross terms to raise the dimension of the original logarithmic domain data to a higher dimension, so as to more precisely describe the characteristics of color distribution. For example, a second-order matrix is calculated based on the first-order matrix to characterize the possible non-linear response differences between sensors. Based on the source domain color feature matrix and the color feature matrix of the target domain sensor in the logarithmic domain, a linear regression algorithm is used to construct an input-output mapping relationship, establish a mapping rule for cross-sensor neutral pixels, and obtain the first conversion model.

[0070] Exemplarily, in order to enable the first conversion model to more effectively capture the complex patterns of the data and improve the accuracy and performance of the first conversion model, from the N paired logarithmic color matrices collected, the log(G / R) and log(G / B) of the 21st color patch of each paired logarithmic color matrix are selected (that is, the logarithmic color matrix of the neutral pixel is selected), so as to obtain a paired color matrix with the shape of [N, 2]. Polynomial feature expansion is performed on the paired color matrix with the shape of [N, 2]. Among them, polynomial feature expansion is implemented through the PolynomialFeatures class in the publicly available sklearn library in Python.

[0071] In this embodiment, polynomial feature expansion is performed on the logarithmic color feature matrix of the source domain sensor, and then based on the expanded source domain color feature matrix, the first conversion model is trained. The expanded source domain color feature matrix can raise the dimension of the data to a higher dimension, enabling the training of the first conversion model to more accurately capture the complex mapping rules across sensors, effectively eliminating the deviation caused by sensor inconsistency, enabling the white balance data applied to the source domain sensor to be directly migrated without re-calibrating for the target domain sensor, reducing the annotation cost of adapting the white balance data to new devices, and realizing the high-efficiency migration of cross-sensor white balance data.

[0072] Optionally, in another embodiment, step S20 includes steps B10 to B20:

[0073] In step B10, the logarithmic domain data of the neutral pixel of the source domain sensor is input into the first conversion model for predicting the illumination mapping, and the predicted logarithmic domain data of the neutral pixel is output.

[0074] It should be noted that the logarithmic domain data of the neutral pixels captured by the source domain sensor is input into the first conversion model, and the logarithmic domain data of the neutral pixels of the target domain sensor under the same illumination conditions is predicted through the mapping relationship of the first conversion model. The first conversion model analyzes the response deviation law of neutral pixels between different sensors and establishes an illumination-independent mapping rule. The first conversion model is used to eliminate the influence of sensor hardware differences on the neutral gray area, provide a reference for cross-device white balance data migration, and ensure the accuracy and consistency of the neutral color temperature estimation of the target domain sensor.

[0075] Step B20: Use the predicted logarithmic domain data of the neutral pixels as the logarithmic domain data of the neutral pixels of the target domain sensor.

[0076] It should be noted that the logarithmic domain data of the neutral pixels predicted by the first conversion model is directly used as the actual logarithmic domain data of the neutral pixels of the target domain sensor. The predicted logarithmic domain data of the neutral pixels is derived through the first conversion model, rather than obtained by physical acquisition relying on the target domain sensor.

[0077] By replacing the actual measurement value of the target domain sensor with the predicted value of the first conversion model, it is ensured that the white balance algorithm parameters of the target domain sensor in the neutral pixel response are strictly consistent with those of the source domain sensor, thereby eliminating the color temperature estimation deviation caused by sensor hardware differences, providing a reference for the subsequent conversion of non-neutral pixels, and finally realizing the lossless cross-device migration of white balance data.

[0078] In this embodiment, by inputting the logarithmic domain data of the neutral pixels of the source domain sensor into the first conversion model, the logarithmic domain data of the neutral pixels is predicted and generated, and it is directly used as the actual logarithmic domain data of the neutral pixels of the target domain sensor to achieve the precise alignment of the neutral gray characteristics across sensors. In addition, based on the illumination mapping model of neutral pixels (the first conversion model), the color temperature deviation of the neutral area caused by sensor hardware differences can be eliminated. Therefore, the white balance data of the source domain sensor can be directly and losslessly migrated to the target domain sensor. At the same time, by replacing the physical acquisition data of the target domain sensor with the predicted value of the first conversion model, the cumbersome process of repeated calibration of the target domain sensor is avoided, and the data calibration cost is significantly reduced.

[0079] Step S30: For the non-neutral pixels in the color chart, use the second conversion model to convert the logarithmic domain data of the non-neutral pixels from the source domain sensor to the target domain sensor.

[0080] It should be noted that the spectral reflectance of non-neutral pixels in the color chart has a specific color tendency and is jointly affected by the color temperature of the light source and the spectral sensitivity of the sensor; the second conversion model is a conversion model based on the mapping result of neutral pixels and the constraint of the Planck curve. The logarithmic domain data of non-neutral pixels of the source domain sensor is converted to the target domain sensor through the second conversion model to achieve cross-sensor color adaptation conversion. By hierarchically compensating for the spectral response differences between sensors, the hue and saturation consistency of non-neutral pixels are ensured during cross-device migration, thereby avoiding color deviation caused by sensor differences.

[0081] In this embodiment, after collecting the color charts of the source domain sensor and the target domain sensor under various lighting conditions, for the neutral pixels in the color chart, illumination mapping is performed through the first conversion model to convert the logarithmic domain data of the neutral pixels from the source domain sensor to the target domain sensor. For the non-neutral pixels in the color chart, the logarithmic domain data of the non-neutral pixels is converted from the source domain sensor to the target domain sensor using the second conversion model. This method does not require re-collecting a large amount of labeled data, can effectively decouple the spectral response differences between different sensors, and enables the logarithmic domain data of neutral pixels and non-neutral pixels in the color chart of the source domain sensor to be used as labeled data. After the labeled data is converted, it can directly adapt to the characteristics of the target domain sensor, improving the efficiency of white balance data during cross-sensor migration while eliminating the need for additional data labeling of the target domain sensor.

[0082] Based on the above embodiments of the present application, in another embodiment of the present application, for the same or similar content as the above embodiments, reference can be made to the above introduction and will not be elaborated hereinafter. Before step S30, it includes steps D10 to D30:

[0083] Step D10, fitting the illumination distributions corresponding to the neutral pixels of the source domain sensor under various illuminations to obtain the Planck curve representing the color temperature distribution of the source domain sensor;

[0084] It should be noted that since the first conversion model only converts the neutral pixels of the image from the source domain sensor to the target domain sensor, there are still significant differences in non-neutral pixels. The purpose of this embodiment is to obtain a second conversion model for converting non-neutral pixels from the source domain sensor to the target domain sensor.

[0085] Using regression algorithms such as the least squares method, the neutral pixels at different color temperatures are fitted to a continuous curve, namely the blackbody locus curve (also known as the Planck curve). This Planck curve characterizes the color distribution law of the ideal blackbody radiation with the change of color temperature in colorimetry. The Planck curve obtained by fitting can accurately describe the actual response characteristics of the source domain sensor to the color temperature. Its role is to provide a color temperature constraint benchmark for the subsequent conversion of non-neutral pixels, ensure the consistency of the color temperature parameters during cross-sensor color transfer, and avoid the color temperature estimation deviation caused by sensor differences.

[0086] Step D20: Map the light distribution corresponding to the non-neutral pixels of the source domain sensor under multiple illuminations to the Planck curve, and obtain the foot point of the logarithmic domain coordinate point of the non-neutral pixel in the coordinate system where the Planck curve is located on the Planck curve.

[0087] It should be noted that for the non-neutral pixels collected by the source domain sensor under multiple color temperature illuminations, project their corresponding light distribution onto the Planck curve representing the color temperature distribution, and calculate the foot point of the non-neutral pixel on the Planck curve through the geometric projection algorithm. Among them, the foot point is the point on the Planck curve closest to the logarithmic domain coordinate point. The foot point is the projection point on the Planck curve closest to the response value of the non-neutral pixel, and its physical meaning is the neutral pixel corresponding to the non-neutral pixel under the current light source color temperature.

[0088] Through step D20, map the sensor response of the non-neutral pixel to the color temperature constraint Planck curve, quantify the color temperature difference value between the non-neutral pixel and the neutral pixel, and provide a reference benchmark with consistent color temperature attributes for the subsequent cross-sensor conversion, so as to eliminate the color temperature estimation deviation caused by the interaction between the light source and the sensor, and ensure that the conversion process of the non-neutral pixel conforms to the colorimetry law.

[0089] Such as Figure 3 shown, Figure 3 is a schematic diagram for mapping the light distribution corresponding to the non-neutral pixels of the source domain sensor under multiple illuminations to the Planck curve, and P is one of the foot points.

[0090] Step D30: Obtain the second conversion model according to the conversion matrix of the foot point and the pre-sampled points adjacent to the foot point.

[0091] It should be noted that based on the foot point of the non-neutral pixel on the Planck curve (i.e., the theoretical neutral pixel projection point of the neutral pixel under color temperature constraint), combined with the conversion matrices of the pre-sampled points adjacent to the foot point (i.e., the points on the left and right of the foot point), a second conversion model is generated through an interpolation algorithm. Specifically, according to the position of the foot point on the color temperature curve, the color temperature difference value between it and the adjacent pre-sampled points is calculated, and the interpolation weights of the conversion matrices of the pre-sampled points are determined accordingly; subsequently, the conversion matrices of the adjacent sampled points are linearly combined according to the weights, and finally the second conversion model is obtained.

[0092] In this embodiment, a Planck curve is generated by fitting the light distribution corresponding to the neutral pixels of the source domain sensor, and a physical constraint benchmark for color temperature distribution is established, so that the color conversion of non-neutral pixels always conforms to the laws of colorimetry; the non-neutral pixels are mapped to the Planck curve and the foot point is calculated to quantify the color temperature correlation between the non-neutral pixels and the theoretical neutral pixels, eliminating the color deviation caused by the light source color temperature; a second conversion model is generated by interpolating the conversion matrices of the foot point and the pre-sampled points to achieve dynamic sensor difference compensation with color temperature adaptability. Through the second conversion model, it is ensured that when non-neutral pixels migrate across sensors, their hue and saturation are accurately adapted synchronously with the change of the light source color temperature, avoiding the problem of color deviation accumulation caused by a fixed conversion matrix in the traditional method.

[0093] Based on the above embodiments of the present application, in another embodiment of the present application, the same or similar content as the above embodiments can be referred to the above introduction and will not be repeated hereinafter. An integration method is provided, and the integration method includes:

[0094] In the data loading stage of training the image processing model of the target domain sensor, a cross-sensor white balance data conversion method is applied to perform cross-sensor white balance data conversion in the data loading stage.

[0095] It should be noted that for the target domain sensor to perform white balance processing, it is necessary to train the image processing model of the target domain sensor, that is, the image processing model of the target domain sensor. The existing training process of the model is divided into: data loading stage, model construction stage, model training stage, and model verification stage. In this embodiment, in the data loading stage of training the image processing model of the target domain sensor, a cross-sensor white balance data conversion method is applied. By calling the first conversion model and the second conversion model in real time, the white balance data applied to the source domain sensor is converted into white balance data that conforms to the distribution law of the target domain sensor. Then, the image processing model of the target domain sensor can be trained using the white balance data that conforms to the distribution law of the target domain sensor. Finally, in the deployment and inference stage of the image processing model, the target domain sensor can use the image processing model to perform white balance processing.

[0096] Performing the conversion during the data loading phase can unify the color response benchmarks of different sensors before the data is input into the model, ensuring that the input data received by the model has sensor-independent color consistency. In this embodiment, the image processing model is not limited.

[0097] Integrate the cross-sensor white balance data conversion algorithm in the data loading module. For each input source domain sensor image, first eliminate the color temperature estimation deviation caused by sensor hardware differences according to the first conversion model for realizing neutral pixel conversion, and then correct the color deviation through the second conversion model for realizing non-neutral pixel conversion, and finally generate data aligned with the color characteristics of the target domain sensor. This method does not require retraining or fine-tuning for different sensors, directly compatible with the input data of multi-sensor devices, and significantly improves the training efficiency of the target domain sensor image processing model.

[0098] In this embodiment, by integrating the cross-sensor white balance data conversion method in the data loading phase of training the image processing model of the target domain sensor, the sensor adaptation and color consistency calibration of the input data are realized. In this way, only a slight increase in the amount of calculation is added in the data loading phase, and the amount of calculation is zero increased in the deployment and inference phase, which can improve the model deployment efficiency.

[0099] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the cross-sensor white balance data conversion method of this application. Based on this technical concept, more forms of simple transformations, such as the interaction and combination of each embodiment, are within the protection scope of this application.

[0100] Exemplarily, the following devices, equipment, storage media and program products can be configured to implement the cross-sensor white balance data conversion method as described above, and / or the steps of the integrated method. The following takes the cross-sensor white balance data conversion method as an example.

[0101] This application also provides a cross-sensor white balance data conversion device, please refer to Figure 4 , the cross-sensor white balance data conversion device includes:

[0102] The acquisition module 10 is used to acquire the color chart images of the source domain sensor and the target domain sensor under various lighting conditions;

[0103] The first conversion module 20 is used to perform light mapping on the neutral pixels in the color chart image through the first conversion model to convert the logarithmic domain data of the neutral pixels from the source domain sensor to the target domain sensor;

[0104] The second conversion module 30 is used to convert the logarithmic domain data of the non-neutral pixels from the source domain sensor to the target domain sensor through the second conversion model for the non-neutral pixels in the color chart image.

[0105] Optionally, the first conversion module 20 is further configured to, under multiple lighting conditions, respectively use the source domain sensor and the target domain sensor to collect images of the same standard color card, so as to obtain multiple pairs of RAW image data;

[0106] Extract the response values of the source domain sensor and the target domain sensor to each color block in the color card from the RAW image data, so as to obtain the color feature matrices of the source domain sensor and the target domain sensor in the logarithmic domain;

[0107] Train a first conversion model through the color feature matrices.

[0108] Optionally, the first conversion module 20 is further configured to perform feature expansion on the color feature matrix of the source domain sensor in the logarithmic domain, so as to obtain an expanded source domain color feature matrix;

[0109] Train a first conversion model through the source domain color feature matrix and the color feature matrix of the target domain sensor in the logarithmic domain.

[0110] Optionally, the first conversion module 20 is further configured to input the logarithmic domain data of the neutral pixels of the source domain sensor into the first conversion model for predicting the light mapping, and output the predicted logarithmic domain data of the neutral pixels.

[0111] Use the predicted logarithmic domain data of the neutral pixels as the logarithmic domain data of the neutral pixels of the target domain sensor.

[0112] Optionally, the second conversion module 30 is further configured to fit the light distribution corresponding to the neutral pixels of the source domain sensor under multiple lighting conditions, so as to obtain a Planck curve representing the color temperature distribution of the source domain sensor;

[0113] Map the light distribution corresponding to the non-neutral pixels of the source domain sensor under multiple lighting conditions to the Planck curve, so as to obtain the foot points of the logarithmic domain coordinate points of the non-neutral pixels in the coordinate system where the Planck curve is located on the Planck curve;

[0114] Obtain a second conversion model according to the conversion matrix of the foot points and the pre-sampled points adjacent to the foot points.

[0115] This application further provides an integrated device, which is used to apply the cross-sensor white balance data conversion method during the data loading stage of training the image processing model of the target domain sensor, so as to perform cross-sensor white balance data conversion during the data loading stage.

[0116] The cross-sensor white balance data conversion device provided by the present application adopts the cross-sensor white balance data conversion method in the above-mentioned embodiment, and can solve the technical problem of low efficiency of white balance data migration across sensors in the prior art. Compared with the prior art, the beneficial effects of the cross-sensor white balance data conversion device provided by the present application are the same as those of the cross-sensor white balance data conversion method provided by the above-mentioned embodiment, and other technical features in the cross-sensor white balance data conversion device are the same as those disclosed in the method of the above-mentioned embodiment, which will not be elaborated here.

[0117] The present application provides a cross-sensor white balance data conversion device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the cross-sensor white balance data conversion method in the above-mentioned first embodiment.

[0118] Refer to the following Figure 5 , which shows a schematic structural diagram of a cross-sensor white balance data conversion device suitable for implementing the embodiments of the present application. The cross-sensor white balance data conversion device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The cross-sensor white balance data conversion device shown is only an example, and should not impose any limitations on the functions and usage scopes of the embodiments of the present application.

[0119] As shown in Figure 5As shown, the cross-sensor white balance data conversion device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. In the random access memory 1004, various programs and data required for the operation of the cross-sensor white balance data conversion device are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. The input / output interface 1006 is also connected to the bus. Generally, the following systems may be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the cross-sensor white balance data conversion device to communicate with other devices wirelessly or wiredly to exchange data. Although the cross-sensor white balance data conversion device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems may be alternatively implemented or had.

[0120] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0121] The cross-sensor white balance data conversion device provided by the present application adopts the cross-sensor white balance data conversion method in the above embodiments, and can solve the technical problem of low efficiency of cross-sensor white balance data migration in the prior art. Compared with the prior art, the beneficial effects of the cross-sensor white balance data conversion device provided by the present application are the same as those of the cross-sensor white balance data conversion method provided by the above embodiments, and other technical features in the cross-sensor white balance data conversion device are the same as the features disclosed in the method of the previous embodiment, and will not be elaborated herein.

[0122] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0123] As described above, the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all of them should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0124] This application provides a computer-readable storage medium with computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the cross-sensor white balance data conversion method in the above embodiments.

[0125] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0126] The above computer-readable storage medium can be included in the cross-sensor white balance data conversion device; it can also exist separately without being assembled into the cross-sensor white balance data conversion device.

[0127] The above computer-readable storage medium carries one or more programs, which, when executed by the cross-sensor white balance data conversion device, cause the cross-sensor white balance data conversion device to: collect color chart images of the source domain sensor and the target domain sensor under various lighting conditions; for the neutral pixels in the color chart image, perform lighting mapping through the first conversion model to convert the logarithmic domain data of the neutral pixels from the source domain sensor to the target domain sensor; for the non-neutral pixels in the color chart image, convert the logarithmic domain data of the non-neutral pixels from the source domain sensor to the target domain sensor through the second conversion model.

[0128] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0130] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0131] The readable storage medium provided by the present application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above cross-sensor white balance data conversion method, which can solve the technical problem of low efficiency of white balance algorithm data migration across sensors in the prior art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the cross-sensor white balance data conversion method provided by the above embodiments, and will not be elaborated here.

[0132] The present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the cross-sensor white balance data conversion method as described above.

[0133] The computer program product provided by the present application can solve the technical problem of low efficiency of white balance data migration across sensors in the prior art. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the cross-sensor white balance data conversion method provided by the above embodiments, and will not be elaborated here.

[0134] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the specification and drawings of the present application under the technical concept of the present application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A method for converting white balance data across sensors, characterized in that: The method comprises: Collect color charts of source domain sensors and target domain sensors under various lighting conditions; For neutral pixels in the color chart image, light mapping is performed through the first conversion model to convert the logarithmic domain data of the neutral pixels from the source domain sensor to the target domain sensor; For non-neutral pixels in the color chart image, the logarithmic domain data of the non-neutral pixels are converted from the source domain sensor to the target domain sensor through a second conversion model.

2. The method according to claim 1, characterized in that Before the step of performing illumination mapping by the first conversion model to convert the logarithmic domain data of the neutral pixel from the source domain sensor to the target domain sensor, the method includes: Under various lighting conditions, the source domain sensor and the target domain sensor are used to collect images of the same standard color card to obtain multiple sets of paired RAW image data; Extracting the response values ​​of the source domain sensor and the target domain sensor to each color block in the color card from the RAW image data, and obtaining the color feature matrix of the source domain sensor and the target domain sensor in the logarithmic domain; The first conversion model is obtained by training the color feature matrix.

3. The method according to claim 2, characterized in that The method further comprises: Perform feature expansion on the color feature matrix of the source domain sensor in the logarithmic domain to obtain the source domain color feature matrix after feature expansion; The first conversion model is obtained by training the source domain color feature matrix and the target domain sensor color feature matrix in the logarithmic domain.

4. The method according to claim 1, characterized in that The step of performing illumination mapping by using the first conversion model to convert the logarithmic domain data of the neutral pixel from the source domain sensor to the target domain sensor comprises: Inputting the logarithmic domain data of the neutral pixel of the source domain sensor into the first conversion model to predict the illumination mapping, and outputting the logarithmic domain data of the predicted neutral pixel; The logarithmic domain data of the predicted neutral pixel is used as the logarithmic domain data of the neutral pixel of the target domain sensor.

5. The method according to claim 1, characterized in that Before the step of converting the logarithmic domain data of the non-neutral pixels from the source domain sensor to the target domain sensor by using the second conversion model, the method includes: The illumination distribution corresponding to the neutral pixel of the source domain sensor under various illuminations is fitted to obtain the Planck curve representing the color temperature distribution of the source domain sensor; Mapping the illumination distribution corresponding to the non-neutral pixel of the source domain sensor under various illuminations to the Planck curve, and obtaining the foot point of the logarithmic domain coordinate point of the non-neutral pixel in the coordinate system of the Planck curve on the Planck curve; A second transformation model is obtained according to the transformation matrix of the perpendicular foot point and the pre-sampling points adjacent to the perpendicular foot point.

6. An integration method, characterized in that: The method further comprises: In the data loading stage of training the image processing model of the target domain sensor, the cross-sensor white balance data conversion method as described in any one of claims 1 to 5 is applied to perform cross-sensor white balance data conversion in the data loading stage.

7. A cross-sensor white balance data conversion device, characterized in that: The cross-sensor white balance data conversion device comprises: An acquisition module, used to acquire color charts of source domain sensors and target domain sensors under various lighting conditions; A first conversion module, configured to perform illumination mapping on neutral pixels in the color chart image through a first conversion model, and convert logarithmic domain data of the neutral pixels from a source domain sensor to a target domain sensor; The second conversion module is used to convert the logarithmic domain data of the non-neutral pixels in the color chart from the source domain sensor to the target domain sensor through the second conversion model.

8. A cross-sensor white balance data conversion device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the cross-sensor white balance data conversion method as described in any one of claims 1 to 5, and / or the steps of the integration method described in claim 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the cross-sensor white balance data conversion method described in any one of claims 1 to 5 and / or the steps of the integration method described in claim 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the cross-sensor white balance data conversion method as described in any one of claims 1 to 5, and / or the steps of the integration method described in claim 6.

Images