Image processing method and device, storage medium, communication equipment and chip

Abstract

The invention provides an image processing method, which comprises the steps of determining a first processing image based on a to-be-calibrated image, the image format of the to-be-calibrated image being a first format, the image format of the first processing image being a second format, the first format being different from the second format, and the to-be-calibrated image being acquired by a first sensor; mapping the first processed image from a first color space of the first sensor to a second color space of the second sensor to determine a first mapped image; performing color space mapping on the first mapping image to determine a second mapping image; and determining a white balance calibration coefficient based on the second mapping image so as to perform white balance calibration on the to-be-calibrated image. According to the method disclosed by the invention, through two times of color space mapping, the white point distribution overlapping degree of the to-be-calibrated image on the plurality of sensors is improved, so that the application range of the image processing method is expanded, and meanwhile, the accuracy of the white balance calibration coefficient is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular, to an image processing method, apparatus, storage medium, communication device, and chip. Background Art

[0002] As an important branch in the field of image processing, white balance is usually used to restore the color of an object in an image acquired by an image sensor to the object's inherent color. In other words, white balance can exclude the influence of the light source color on the color of the image object.

[0003] With the development of technology, more and more devices are equipped with multiple sensors. Therefore, white balance across sensors has become a hot topic among major manufacturers. However, due to the differences in the responses of different types of image sensors to different colors to a certain extent, a white balance method that can achieve cross-sensor and low computing power overhead is required. Summary of the Invention

[0004] The present disclosure provides an image processing method, apparatus, storage medium, communication device, and chip to solve the problems in the related art, and can achieve cross-sensor image white balance, while improving the applicable range of the cross-sensor image white balance method and the accuracy of the white balance calibration coefficient.

[0005] In a first aspect embodiment of the present disclosure, an image processing method is proposed. The method includes: based on a to-be-calibrated image, determining a first processed image, where the image format of the to-be-calibrated image is a first format, the image format of the first processed image is a second format, the first format is different from the second format, and the to-be-calibrated image is acquired by a first sensor; mapping the first processed image from the first color space of the first sensor to the second color space of a second sensor to determine a first mapped image; performing a color space mapping on the first mapped image to determine a second mapped image; and based on the second mapped image, determining a white balance calibration coefficient to perform white balance calibration on the to-be-calibrated image.

[0006] In some embodiments, determining a first processed image based on the to-be-calibrated image includes: preprocessing the to-be-calibrated image, where the preprocessing at least includes: shadow correction processing and black level correction processing; and based on the preprocessed to-be-calibrated image, determining the first processed image.

[0007] In some embodiments, using a first color transformation matrix to perform a first color space mapping on the first processed image to determine a first mapped image includes: determining a first color card pixel value of the first sensor and a second color card pixel value of the second sensor; based on the first color card pixel value and the second color card pixel value, determining the first color transformation matrix; and based on the first processed image and the first color transformation matrix, determining the first mapped image.

[0008] In some embodiments, performing color space mapping on a first mapped image to determine a second mapped image includes: determining first chromaticity histogram features of the first mapped image; determining a second color transformation matrix based on the first chromaticity logarithmic histogram features; and determining the second mapped image based on the first mapped image and the second color transformation matrix.

[0009] In some embodiments, determining a white balance calibration coefficient based on the second mapped image to perform white balance calibration on an image to be calibrated includes: determining white point estimation parameters of the image to be calibrated based on the second mapped image; determining the white balance calibration coefficient based on the white point estimation parameters, a first color transformation matrix, and the second color transformation matrix; and performing white balance calibration on the image to be calibrated by using the white balance calibration coefficient.

[0010] In some embodiments, determining the white point estimation parameters of the image to be calibrated based on the second mapped image includes: determining second chromaticity histogram features of the second mapped image; and determining the white point estimation parameters of the image to be calibrated based on the second chromaticity histogram features.

[0011] An embodiment of the second aspect of the present disclosure provides an image processing apparatus, including: a first processing unit configured to determine a first processed image based on an image to be calibrated, where the image format of the image to be calibrated is a first format, the image format of the first processed image is a second format, the first format is different from the second format, and the image to be calibrated is acquired by a first sensor; a second processing unit configured to map the first processed image from a first color space of the first sensor to a second color space of a second sensor to determine a first mapped image; a third processing unit configured to perform color space mapping on the first mapped image to determine a second mapped image; and a fourth processing unit configured to determine a white balance calibration coefficient based on the second mapped image to perform white balance calibration on the image to be calibrated.

[0012] An embodiment of the third method aspect of the present disclosure provides a computer-readable storage medium, where when a computer program is executed by a processor, the method described in the embodiment of the first aspect of the present disclosure is performed.

[0013] An embodiment of the fourth aspect of the present disclosure provides a communication device, including: a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, where when the processor executes the computer program, the method described in the first aspect of the embodiments of the present disclosure is performed.

[0014] An embodiment of the fifth aspect of the present disclosure provides a chip, including one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of a communication device and send the signal to the processor, the signal includes computer instructions stored in the memory, and when the processor executes the computer instructions, the communication device is caused to perform the method described in the embodiment of the first aspect of the present disclosure.

[0015] In summary, according to the image processing method proposed by the present disclosure, the method includes: determining a first processed image based on an image to be calibrated, where the image format of the image to be calibrated is a first format, the image format of the first processed image is a second format, the first format is different from the second format, and the image to be calibrated is acquired by a first sensor; mapping the first processed image from the first color space of the first sensor to the second color space of a second sensor to determine a first mapped image; performing a color space mapping on the first mapped image to determine a second mapped image; and determining a white balance calibration coefficient based on the second mapped image to perform white balance calibration on the image to be calibrated. The method of the present disclosure improves the overlap degree of white point distributions of the image to be calibrated on multiple sensors through two color space mappings, thereby expanding the applicable range of the present image processing method and improving the accuracy of the white balance calibration coefficient at the same time.

[0016] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure and do not constitute an improper limitation of the present disclosure.

[0018] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0019] Figure 2 It is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0020] Figure 3 It is an example diagram of an image processing method provided by an embodiment of the present disclosure;

[0021] Figure 4 It is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0022] Figure 5 It is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0023] Figure 6 It is an example diagram of an image processing method provided by an embodiment of the present disclosure;

[0024] Figure 7 It is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present disclosure;

[0025] Figure 8 It is a schematic structural diagram of a communication device provided by an embodiment of the present disclosure;

[0026] Figure 9 Schematic diagram of the structure of a chip provided by an embodiment of the present disclosure. Detailed implementation manners

[0027] The embodiments of the present disclosure will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present disclosure, but should not be construed as a limitation to the present disclosure.

[0028] As an important branch in the field of image processing, white balance is usually used to restore the color of an object in an image acquired by an image sensor to the object's original color. In other words, white balance can exclude the influence of the light source color on the color of the image object.

[0029] With the development of technology, more and more devices are equipped with multiple sensors. Therefore, white balance across sensors has become a hot topic of concern for major manufacturers. However, due to the differences in the responses of different types of image sensors to different colors to a certain extent, a white balance method that can achieve cross-sensor and low computing power overhead is needed.

[0030] In the related art, a white balance method is to perform color space alignment by regressing the color conversion matrix of an image, so as to achieve cross-sensor image white balance. However, when the spectral response differences between different types of sensors are large, the white balance accuracy of this method will decrease, so the applicability of this method is limited.

[0031] Another white balance method is to use a super network to learn information such as the sensitivity of different sensors, and use the fast Fourier transform to replace the convolution calculation of the white point information, so as to achieve cross-sensor image white balance. However, in the learning process of different sensors, a large number of supervised learning data are required, the practical difficulty is great, and the model computing power is relatively large, which is not conducive to the deployment of low-computing-power intelligent mobile terminals.

[0032] Therefore, to solve the problems existing in the related art, the present disclosure proposes an image processing method. By performing two color space mappings, the overlap degree of the white point distributions of the image to be calibrated on multiple sensors is improved, thereby expanding the applicable range of the present image processing method and at the same time improving the accuracy of the white balance calibration coefficient.

[0033] Figure 1 Flowchart of an image processing method provided by an embodiment of the present disclosure. This method can be applicable to Figure 1 the application scenarios shown, for example, executed by a terminal integrated with image processing functions or an image processor in the terminal, which is not limited by the present disclosure. As Figure 1 shown, Figure 1It may include the following steps.

[0034] Step 101: Based on the image to be calibrated, determine a first processed image.

[0035] In some embodiments, the image to be calibrated is acquired by a first sensor, where the first sensor can be any one of multiple sensors in the device that executes this method.

[0036] In some embodiments, based on the image to be calibrated, determine a first processed image to lay a foundation for the first color space mapping.

[0037] Optionally, the first processed image can be determined by preprocessing the image to be calibrated and converting the image format of the preprocessed image to be calibrated. The image format of the image to be calibrated is a first format, and the image format of the first processed image is a second format.

[0038] In some embodiments, the first format is the Bayer format, and the second format is the RGB format.

[0039] In some embodiments, the image format of the image to be calibrated acquired by the first sensor is the Bayer format. Therefore, it is necessary to convert the image format of the image to be calibrated into an RGB image format that can perform color space mapping. In other words, the image to be calibrated is the original image acquired by the first sensor, and the first processed image is the image of the image to be calibrated after color space mapping.

[0040] Step 102: Map the first processed image from the first color space of the first sensor to the second color space of the second sensor to determine a first mapped image.

[0041] In some embodiments, mapping the first processed image from the first color space of the first sensor to the second color space of the second sensor can enable the determination of white balance coefficients for the first processed images acquired by different first sensors in the same mapping space, improving the applicability of the method of the present disclosure.

[0042] In some embodiments, it is possible to determine the first color card pixel values of the first sensor and the second color card pixel values of the second sensor; and determine a first color transformation matrix based on the first color card pixel values and the second color card pixel values; thereby using the first color transformation matrix to perform color space mapping on the first processed image to determine the first mapped image.

[0043] In some embodiments, the first sensor is the sensor that acquires the image to be calibrated, and the second sensor can be any sensor other than the first sensor in the device that executes this method.

[0044] Step 103: Perform color space mapping on the first mapped image to determine the second mapped image.

[0045] In some embodiments, by performing color space mapping on the first mapped image, the dispersion degree of the white point of the first mapped image in the mapping space is further reduced, thereby further improving the applicable range of the method of the present disclosure and the accuracy of the white balance coefficient.

[0046] In some embodiments, the first chromaticity histogram feature of the first mapped image can be determined; and the second color transformation matrix can be determined through the first chromaticity logarithmic histogram feature; thereby, the second color transformation matrix is used to perform color space mapping on the first mapped image to determine the second mapped image.

[0047] Step 104: Based on the second mapped image, determine the white balance calibration coefficient to perform white balance calibration on the image to be calibrated.

[0048] In some embodiments, the white point estimation parameter of the image to be calibrated can be determined based on the second mapped image, and then the white balance calibration coefficient can be determined by using the white point estimation parameter, the first color transformation matrix, and the second color transformation matrix; finally, the white balance calibration coefficient is used to perform white balance calibration on the image to be calibrated to achieve white balance calibration of the image to be calibrated.

[0049] In summary, according to the image processing method proposed by the present disclosure, the method includes: based on the image to be calibrated, determine the first processed image, the image format of the image to be calibrated is the first format, the image format of the first processed image is the second format, the first format is different from the second format, and the image to be calibrated is acquired by the first sensor; map the first processed image from the first color space of the first sensor to the second color space of the second sensor to determine the first mapped image; perform color space mapping on the first mapped image to determine the second mapped image; based on the second mapped image, determine the white balance calibration coefficient to perform white balance calibration on the image to be calibrated. The method of the present disclosure improves the overlap degree of the white point distribution of the image to be calibrated on multiple sensors through two color space mappings, thereby expanding the applicable range of the present image processing method and improving the accuracy of the white balance calibration coefficient at the same time.

[0050] Figure 2 Further shows the flowchart of an image processing method proposed by the present disclosure. Based on Figure 2 the embodiments shown, step 101 is further explained, Figure 2 it can include the following steps.

[0051] Step 201: Preprocess the image to be calibrated.

[0052] In some embodiments, by preprocessing the image to be calibrated, factors such as luma shading, color shading, and dark current that affect the image quality of the image to be calibrated are removed to improve the accuracy of the white balance coefficient.

[0053] In some embodiments, the preprocessing at least includes shadow correction processing and black level correction processing. The present disclosure does not limit the specific manner of the shadow correction processing. For example: grid method, concentric circle method, etc. The present disclosure does not limit the specific manner of the black level correction processing. For example: curve fitting method, fixed value deduction method, etc.

[0054] Step 202: Based on the preprocessed image to be calibrated, determine the first processed image.

[0055] In some embodiments, the image format of the preprocessed image to be calibrated is the Bayer format. Therefore, it is necessary to convert the image format of the preprocessed image to be calibrated into the RGB format that can perform color space mapping.

[0056] Optionally, the R-channel data and B-channel data in the preprocessed image to be calibrated can be determined as the R-channel data and B-channel data of the first processed image, and the Gr-channel data and Gb-channel data in the preprocessed image to be calibrated are averaged, and the average value is determined as the G-channel data of the first processed image, thereby realizing the determination of the first processed image.

[0057] In summary, according to the image processing method proposed by the present disclosure, the method includes: preprocessing the image to be calibrated, and the preprocessing at least includes: shadow correction processing and black level correction processing; based on the preprocessed image to be calibrated, determine the first processed image. The method of the present disclosure preprocesses the image to be calibrated to remove the influence of factors such as vignetting, color shading, and dark current on the image quality of the image to be calibrated, and converts the image format of the preprocessed image to be calibrated into an image format that can perform color space mapping through image format conversion.

[0058] Figure 3 Further shows a flowchart of an image processing method proposed by the present disclosure. Based on Figure 1 the embodiments shown, step 102 is further explained. Figure 3 It may include the following steps.

[0059] Step 301: Determine the first color card pixel value of the first sensor and the second color card pixel value of the second sensor.

[0060] In some embodiments, the first color card pixel value and the second color card pixel value may be 24-color card pixel values.

[0061] In some embodiments, the first color card pixel value and the second color card pixel value of the second sensor can be determined based on parameters such as the spectral reflectance of the sensor, the spectral response sensitivity, and the spectral power density of the light source, but are not limited thereto. The present disclosure does not limit the manner of determining the first color card pixel value and the second color card pixel value.

[0062] Exemplarily, the first color card pixel value and the second color card pixel value can be determined by the following formula:

[0063]

[0064] Wherein, I ref represents the first color card pixel value, R λ,k represents the spectral reflectance of the 24-color card of the first sensor under E light, represents the spectral response sensitivity of the first sensor, and L λ represents the spectral power density of the light source.

[0065]

[0066] Wherein, I cur represents the second color card pixel value, r′ λ,k represents the second R λ,k represents the spectral reflectance of the 24-color card of the first sensor under E light, represents the waterfall viewing response sensitivity of the first sensor.

[0067] Step 302: Determine the first color transformation matrix based on the first color card pixel value and the second color card pixel value.

[0068] In some embodiments, the first color transformation matrix can be determined based on the first color card pixel value and the second color card pixel value to perform color space mapping on the first processed image using the first color transformation matrix. In other words, the first color transformation matrix is used to map the first processed image from the first color space to the second color space.

[0069] Exemplarily, the first color transformation matrix can be determined by the following formula:

[0070]

[0071] Wherein, CTM 3×3 represents the first color transformation matrix, and the matrix size of the first color transformation matrix is 3×3, T represents transpose, and -1 represents the inverse of the matrix. It should be understood that this is only an example here, and the present disclosure does not limit the matrix size of the first color transformation matrix.

[0072] Step 303: Determine the first mapped image based on the first processed image and the first color transformation matrix.

[0073] In some embodiments, the first mapped image can be determined by multiplying the first processed image by the first color transformation matrix, so as to map the first processed image from the first color space to the second color space.

[0074] Exemplarily, the first mapped image can be determined by the following formula:

[0075] RGB ctm = RGB · CTM (Equation 4)

[0076] where RGB ctm represents the first mapped image, RGB ctm represents the first mapped image, and CTM represents the first color transformation matrix.

[0077] In summary, the image processing method proposed according to the present disclosure includes: determining the first color card pixel value of the first sensor and the second color card pixel value of the second sensor; determining the first color transformation matrix based on the first color card pixel value and the second color card pixel value; determining the first mapped image based on the first processed image and the first color transformation matrix. The method of the present disclosure determines the first color card pixel value and the second color card pixel value, and then uses the above color card pixel values to determine the first color transformation matrix required for mapping the first processed image from the first color space to the second color space, so as to realize the spatial mapping of the first processed image, so that the first processed images obtained by different first sensors can determine the white balance coefficient in the same mapped space, improving the applicable range of the method of the present disclosure.

[0078] Figure 4 Further, a flowchart of an image processing method proposed by the present disclosure is shown. Based on Figure 1 the embodiments shown, step 103 is further explained, Figure 4 which may include the following steps.

[0079] Step 401, determining the first chromaticity histogram feature of the first mapped image.

[0080] In some embodiments, by determining the first chromaticity histogram feature to determine the second mapped image using the first chromaticity histogram feature, compared with directly using the first mapped image (i.e., directly mapping the RGB format image) to determine the second mapped image, the computing power consumption of the device executing this method can be reduced, so that this method can be applicable to small computing power devices, further expanding the applicable range of this method.

[0081] In some embodiments, the first chromaticity logarithmic histogram feature can be determined by the following formula:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] Among them, I R represents the R-channel pixel value of the first pixel point in the first processed image, and the first pixel point is any pixel point in the first processed image, I B represents the B-channel pixel value of the first pixel point, I G represents the G-channel pixel value of the first pixel point, I y represents the pixel brightness modulus length of the first pixel point, I u1 、I u2 、I u3 、I v1 、I v2 and I v3 represent six chromaticity logarithm information of the first pixel point, H I represents the first chromaticity logarithm histogram feature of the first pixel point, ε is a preset parameter, which is usually a small positive constant, σ is a preset parameter, and u and v represent the range of logarithmic space coordinates.

[0091] Optionally, as shown in Equation 13, the first chromaticity logarithm histogram feature can also be normalized to facilitate model learning by the device implementing the method of the present disclosure, represents the normalized first chromaticity logarithm histogram feature, and norm represents the normalization process.

[0092]

[0093] Step 402: Determine a second color transformation matrix based on the first chromaticity logarithm histogram feature.

[0094] In some embodiments, the second color transformation matrix can be determined by performing grouped convolution and point convolution processing on the first chromaticity logarithm histogram feature of the first mapped image, and using a feature pyramid to extract features from the data obtained by the grouped convolution and point convolution.

[0095] It should be understood that the matrix size of the second color transformation matrix should be the same as that of the first color transformation matrix.

[0096] Step 403: Determine a second mapped image based on the first mapped image and the second color transformation matrix.

[0097] In some embodiments, the second mapped image can be determined by multiplying the first mapped image by the second color transformation matrix to perform color space mapping on the first mapped image, thereby further reducing the white point dispersion degree of the first mapped image in the mapping space, and further improving the applicable range of the method of the present disclosure and the accuracy of the white balance coefficient.

[0098] Exemplarily, the second mapped image can be determined by the following formula:

[0099] RGB net = RGB ctm ·CTM net (Equation 14)

[0100] where RGB net represents the second mapped image, and CTM net represents the second color transformation matrix.

[0101] In summary, the image processing method proposed according to the present disclosure includes: determining the first chromaticity histogram feature of the first mapped image; determining the second color transformation matrix based on the first chromaticity logarithmic histogram feature; and determining the second mapped image based on the first mapped image and the second color transformation matrix. By determining the first chromaticity histogram feature, the method of the present disclosure performs color space mapping on the first mapped image using the first chromaticity histogram feature. Compared with directly determining the second mapped image using the first mapped image (i.e., directly mapping an RGB format image), the computing power consumption of the device executing this method can be reduced, so that this method can be applicable to small computing power devices, further expanding the applicable range of this method; at the same time, by determining the second mapped image, color space mapping of the first mapped image is achieved, thereby further reducing the white point dispersion degree of the first mapped image in the mapping space, and further improving the applicable range of the method of the present disclosure and the accuracy of the white balance coefficient.

[0102] Figure 5 Further shows a flowchart of an image processing method proposed by the present disclosure. Based on Figure 1 the embodiments shown, step 104 is further explained, Figure 5 which may include the following steps.

[0103] Step 501: Determine the white point estimation parameter of the image to be calibrated based on the second mapped image.

[0104] In some embodiments, the white point estimation parameters of the image to be calibrated can be determined based on the second mapped image, laying a foundation for determining the white balance calibration coefficients.

[0105] In some embodiments, the second chromatic histogram features of the second mapped image can be determined; and then, based on the second chromatic histogram features, the white point estimation parameters of the image to be calibrated can be determined.

[0106] Among them, the determination method of the second chromatic histogram features is the same as the method shown in step 401, and reference can be made to the relevant description of the embodiment shown in step 401, which will not be elaborated here.

[0107] Furthermore, the second chromatic logarithmic histogram features of the second mapped image can be subjected to grouped convolution and point convolution processing, and the data obtained from the grouped convolution and point convolution can be subjected to feature extraction using a feature pyramid. Then, through full convolution, the extracted features are mapped into a feature vector, and this feature vector is the white point estimation parameter.

[0108] Step 502: Determine the white balance calibration coefficients based on the white point estimation parameters, the first color transformation matrix, and the second color transformation matrix.

[0109] In some embodiments, the first color transformation matrix and the second color transformation matrix can be used to restore the mapping space of the white point estimation parameters to the first color space of the first sensor, thereby determining the white balance calibration coefficients.

[0110] Optionally, the white balance calibration coefficients can be determined by the following formula:

[0111]

[0112] Among them, WP represents the white balance calibration coefficient, and WP ctm represents the white point estimation parameter.

[0113] Specifically, the white balance calibration coefficient can be expressed in the following form

[0114] WP = {rg, 1, bg} (Equation 16)

[0115] Among them, rg represents the white balance calibration coefficient of the R channel in the image to be calibrated, and bg represents the white balance calibration coefficient of the B channel in the image to be calibrated.

[0116] Step 503: Perform white balance calibration on the image to be calibrated using the white balance calibration coefficients.

[0117] In some embodiments, the image to be calibrated can be white balance calibrated by the following formula:

[0118]

[0119] Among them, S r represents the R-channel data of the image to be calibrated, and S Gr represents the Gr-channel data of the image to be calibrated, and S Gb represents the Gb-channel data of the image to be calibrated, and S B represents the B-channel data of the image to be calibrated. R represents the channel data of the R channel of the image after white balance calibration, Gr represents the channel data of the Gr channel of the image after white balance calibration, Gb represents the channel data of the Gb channel of the image after white balance calibration, and B represents the channel data of the G channel of the image after white balance calibration.

[0120] Figure 6 FIG. is a flow example diagram of an image processing provided by an embodiment of the present disclosure, Figure 6 which may include the following steps.

[0121] 1. First, preprocess the bayer data of the sensor that acquires the image to be calibrated, and output a three-channel linear RGB image (i.e., the above-mentioned first processed image). The preprocessing mainly includes:

[0122] Perform lens shading correction and black level reduction on the bayer data to reduce the influence of vignetting, color shading, and dark current on the quality of the model input image;

[0123] Convert the bayer data into an RGB three-channel image: the R and B channels remain unchanged, and the Gr and Gb are averaged to obtain the G channel.

[0124] Then perform a first-level color space mapping process on the three-channel RGB image to obtain RGB ctm (i.e., the first mapped image). The specific process is as follows:

[0125] 1) Select a reference sensor among multiple different types of image sensors, and calculate the 24-color card pixel values of the sensor.

[0126] 2) Calculate the 24-color card pixel values of the sensor that acquires the image to be calibrated.

[0127] 3) Map the current sensor image to the reference sensor color space to obtain RGB ctm .

[0128] 2. By using RGB ctm calculate the chromaticity logarithmic histogram and use as the input of the color conversion matrix regression network to obtain CTM net , where the color conversion matrix regression network uses grouped convolution and point convolution to reduce the model computing power, and uses a feature pyramid structure to obtain CTM net (CTM net(is a 3x3 matrix).

[0129] 3. Use CTM net for RGB ctm to perform a secondary color space mapping to obtain RGB net (i.e., the above-mentioned second mapped image),

[0130] 4. Extract the chromaticity logarithm feature map of RGB net and use the chromaticity logarithm feature map of RGB net as the input of the white point estimation network. Among them, the white point estimation network uses grouped convolution and point convolution to reduce the computing power of the model, and uses a feature pyramid structure to obtain the image features of RGB net Finally, the white point estimation result WP in the secondary mapped color space is output through a fully convolutional layer net .

[0131] 5. Perform an inverse color space mapping on the white point estimation result to obtain the white point estimation result of the current sensor input data.

[0132] 6. Use the calculated white point estimation result to perform white balance correction on the bayer data input by the sensor.

[0133] Therefore, the present method has the following beneficial effects:

[0134] 1. The method of the present disclosure improves the overlap degree of white point distributions of the images to be calibrated on multiple sensors through two color space mappings, thereby expanding the applicable range of the present image processing method and improving the accuracy of the white balance calibration coefficient at the same time.

[0135] 2. The method of the present disclosure determines the first color card pixel value and the second color card pixel value, and then uses the above-mentioned color card pixel values to determine the first color transformation matrix required for mapping the first processed image from the first color space to the second color space to realize the spatial mapping of the first processed image, so that the first processed images obtained by different first sensors can determine the white balance coefficients in the same mapping space, improving the applicable range of the method of the present disclosure.

[0136] 3. The method of the present disclosure determines the first chromaticity histogram feature to perform color space mapping on the first mapped image using the first chromaticity histogram feature. Compared with directly using the first mapped image (i.e., directly mapping an RGB format image) to determine the second mapped image, it can reduce the computing power consumption of the device executing this method, so that this method can be applicable to small computing power devices, further expanding the applicable scope of this method. At the same time, by determining the second mapped image, color space mapping of the first mapped image is realized, thereby further reducing the white point dispersion degree of the first mapped image in the mapping space, and further improving the applicable scope of the method of the present disclosure and the accuracy of the white balance coefficient.

[0137] Figure 7 FIG. 4 is a schematic structural diagram of an image processing apparatus 700 provided by an embodiment of the present disclosure. The image quality apparatus includes:

[0138] A first processing unit 710, configured to determine a first processed image based on an image to be calibrated. The image format of the image to be calibrated is a first format, and the image format of the first processed image is a second format. The first format is different from the second format. The image to be calibrated is acquired by a first sensor.

[0139] A second processing unit 720, configured to map the first processed image from a first color space of a first sensor to a second color space of a second sensor to determine a first mapped image.

[0140] A third processing unit 730, configured to perform color space mapping on the first mapped image to determine a second mapped image.

[0141] A fourth processing unit 740, configured to determine a white balance calibration coefficient based on the second mapped image to perform white balance calibration on the image to be calibrated.

[0142] In some embodiments, the first processing unit 710 is further configured to perform preprocessing on the image to be calibrated. The preprocessing at least includes: shadow correction processing and black level correction processing; and determine the first processed image based on the preprocessed image to be calibrated.

[0143] In some embodiments, the second processing unit 720 is further configured to determine a first color card pixel value of the first sensor and a second color card pixel value of the second sensor; determine a first color transformation matrix based on the first color card pixel value and the second color card pixel value; and determine the first mapped image based on the first processed image and the first color transformation matrix.

[0144] In some embodiments, the third processing unit 730 is further configured to determine a first chromaticity histogram feature of the first mapped image; determine a second color transformation matrix based on the first chromaticity logarithmic histogram feature; and determine the second mapped image based on the first mapped image and the second color transformation matrix.

[0145] In some embodiments, the fourth processing unit 740 is further configured to determine the white point estimation parameter of the image to be calibrated based on the second mapped image; determine the white balance calibration coefficient based on the white point estimation parameter, the first color transformation matrix, and the second color transformation matrix; and perform white balance calibration on the image to be calibrated by using the white balance calibration coefficient.

[0146] In some embodiments, the fourth processing unit 740 is further configured to determine the second chromaticity histogram feature of the second mapped image; and determine the white point estimation parameter of the image to be calibrated based on the second chromaticity histogram feature.

[0147] According to the image processing apparatus provided by the present disclosure, a first processed image is determined based on an image to be calibrated, the image format of the image to be calibrated is a first format, the image format of the first processed image is a second format, the first format is different from the second format, and the image to be calibrated is acquired by a first sensor; the first processed image is mapped from the first color space of the first sensor to the second color space of the second sensor to determine a first mapped image; the first mapped image is subjected to a color space mapping to determine a second mapped image; and a white balance calibration coefficient is determined based on the second mapped image to perform white balance calibration on the image to be calibrated. The apparatus of the present disclosure improves the overlap degree of the white point distributions of the image to be calibrated on multiple sensors through two color space mappings, thereby expanding the applicable range of the present image processing method and improving the accuracy of the white balance calibration coefficient at the same time.

[0148] Since the apparatus provided in the embodiments of the present disclosure corresponds to the methods provided in the above several embodiments, the implementation manners of the methods are also applicable to the apparatus provided in the present embodiment and will not be described in detail in the present embodiment.

[0149] In the above embodiments provided by the present application, the methods and apparatuses provided by the embodiments of the present application are introduced. To implement the various functions in the methods provided by the embodiments of the present application, a communication device may include a hardware structure and software modules, and implement the above various functions in the form of a hardware structure, a software module, or a combination of a hardware structure and a software module. A certain function among the above various functions may be executed in the form of a hardware structure, a software module, or a combination of a hardware structure and a software module.

[0150] Figure 8 FIG. 18 is a schematic structural diagram of a communication device 800 provided by an embodiment of the present application. The communication device 800 may be a network device, a terminal device, a chip, a chip system, or a processor that supports the network device to implement the above method, or a chip, a chip system, or a processor that supports the terminal device to implement the above method. The apparatus may be used to implement the method described in the above method embodiments, and specific reference may be made to the description in the above method embodiments.

[0151] The communication device 800 may include one or more processors 801. The processor 801 may be a general-purpose processor or a dedicated processor, etc. For example, it may be a baseband processor or a central processing unit. The baseband processor may be used to process the communication protocol and communication data, and the central processing unit may be used to control the communication device (such as a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute a computer program, and process the data of the computer program.

[0152] Optionally, the communication device 800 may further include one or more memories 802, on which a computer program 804 may be stored, and the processor 801 executes the computer program 804 so that the communication device 800 performs the method described in the above method embodiment. Optionally, data may also be stored in the memory 802. The communication device 800 and the memory 802 may be provided separately or integrated together.

[0153] Optionally, the communication device 800 may further include a transceiver 805 and an antenna 806. The transceiver 805 may be referred to as a transceiver unit, a transceiver, or a transceiver circuit, etc., and is used to implement a transceiver function. The transceiver 805 may include a receiver and a transmitter, the receiver may be referred to as a receiver or a receiving circuit, etc., and is used to implement a receiving function; the transmitter may be referred to as a transmitter or a transmitting circuit, etc., and is used to implement a transmitting function.

[0154] Optionally, the communication device 800 may further include one or more interface circuits 807. The interface circuit 807 is used to receive code instructions and transmit them to the processor 801. The processor 801 executes the code instructions to enable the communication device 800 to execute the method described in the above method embodiment.

[0155] In one implementation, the processor 801 may include a transceiver for implementing receiving and sending functions. For example, the transceiver may be a transceiver circuit, an interface, or an interface circuit. The transceiver circuit, interface, or interface circuit for implementing the receiving and sending functions may be separate or integrated. The above-mentioned transceiver circuit, interface, or interface circuit may be used for reading and writing code / data, or the above-mentioned transceiver circuit, interface, or interface circuit may be used for transmitting or delivering signals.

[0156] In one implementation, the processor 801 may store a computer program 803, which runs on the processor 801 and enables the communication device 800 to perform the method described in the above method embodiment. The computer program 803 may be fixed in the processor 801, in which case the processor 801 may be implemented by hardware.

[0157] In one implementation, the communication device 800 may include circuitry that can implement the functions of transmitting, receiving, or communicating in the foregoing method embodiments. The processors and transceivers described in this application may be implemented on an integrated circuit (IC), analog IC, radio frequency integrated circuit (RFIC), mixed-signal IC, application specific integrated circuit (ASIC), printed circuit board (PCB), electronic device, etc. The processors and transceivers may also be fabricated using various IC process technologies, such as complementary metal oxide semiconductor (CMOS), N-type metal oxide semiconductor (NMOS), P-type metal oxide semiconductor (PMOS), bipolar junction transistor (BJT), BiCMOS, silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0158] The communication device described in the above embodiments may be a network device or a terminal device, but the scope of the communication device described in this application is not limited thereto, and the structure of the communication device may not be limited by Figure 8 . The communication device may be an independent device or may be part of a larger device. For example, the communication device may be:

[0159] (1) An independent integrated circuit (IC), or chip, or chip system or subsystem;

[0160] (2) A collection of one or more ICs, optionally, the IC collection may also include storage components for storing data and computer programs;

[0161] (3) An ASIC, such as a modem;

[0162] (4) A module that can be embedded in other devices;

[0163] (5) A receiver, terminal device, smart terminal device, cellular phone, wireless device, handset, mobile unit, vehicle-mounted device, network device, cloud device, artificial intelligence device, etc.;

[0164] (6) Others, etc.

[0165] For the case where the communication device may be a chip or a chip system, reference may be made toFigure 9 Schematic structural diagram of the chip shown.

[0166] An embodiment of the present disclosure also provides a chip, such as Figure 9 the chip shown includes a processor 901 and an interface 902. Among them, the number of processors 901 can be one or more, and the number of interfaces 902 can be multiple.

[0167] Optionally, the chip further includes a memory 903, and the memory 903 is used to store necessary computer programs and data.

[0168] An embodiment of the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause a computer to execute the method described in the above embodiments of the present disclosure.

[0169] Those skilled in the art can also understand that the various illustrative logical blocks and steps listed in the embodiments of the present application can be implemented by electronic hardware, computer software, or a combination of both. Whether such a function is implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art can use various methods to implement the functions for each specific application, but such implementation should not be construed as exceeding the scope protected by the embodiments of the present application.

[0170] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0171] In the description of this specification, the descriptions referring to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0172] Any process or method description represented in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations, where functions may be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0173] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing a logical function, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with such instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (control method), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, a computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0174] It should be understood that various parts of the embodiments of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0175] Those of ordinary skill in the art can understand that all or part of the steps carried out in the method of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0176] In addition, in each of the embodiments of the present invention, each functional unit can be integrated in a processing module, or each unit can exist physically alone, or two or more units can be integrated in one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disc, etc.

[0177] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that, The method includes: Based on the image to be calibrated, determining a first processed image, where the image format of the image to be calibrated is a first format, the image format of the first processed image is a second format, the first format is different from the second format, and the image to be calibrated is acquired by a first sensor; Mapping the first processed image from the first color space of the first sensor to the second color space of a second sensor to determine a first mapped image; Performing color space mapping on the first mapped image to determine a second mapped image; Based on the second mapped image, determining a white balance calibration coefficient to perform white balance calibration on the image to be calibrated.

2. The method according to claim 1, wherein The determining the first processed image based on the image to be calibrated includes: Performing preprocessing on the image to be calibrated, where the preprocessing at least includes: shadow correction processing and black level correction processing; Based on the preprocessed image to be calibrated, determining the first processed image.

3. The method according to claim 1, characterized in that, The using a first color transformation matrix to perform first color space mapping on the first processed image to determine a first mapped image includes: Determining the first color card pixel values of the first sensor and the second color card pixel values of the second sensor; Based on the first color card pixel values and the second color card pixel values, determining a first color transformation matrix; Based on the first processed image and the first color transformation matrix, determining a first mapped image.

4. The method according to claim 1, wherein The performing color space mapping on the first mapped image to determine a second mapped image includes: Determining the first chromaticity histogram features of the first mapped image; Based on the first chromaticity logarithmic histogram features, determining a second color transformation matrix; Based on the first mapped image and the second color transformation matrix, determining a second mapped image.

5. The method according to claim 1, characterized in that The based on the second mapped image, determining a white balance calibration coefficient to perform white balance calibration on the image to be calibrated includes: Based on the second mapped image, determining the white point estimation parameters of the image to be calibrated; Based on the white point estimation parameters, the first color transformation matrix, and the second color transformation matrix, determining a white balance calibration coefficient; Using the white balance calibration coefficient to perform the white balance calibration on the image to be calibrated.

6. The method according to claim 5, wherein The based on the second mapped image, determining the white point estimation parameters of the image to be calibrated includes: Determining the second chromaticity histogram features of the second mapped image; Based on the second chromaticity histogram features, determining the white point estimation parameters of the image to be calibrated.

7. An image processing apparatus, characterized in that, Includes: A first processing unit, configured to determine a first processed image based on an image to be calibrated, where the image format of the image to be calibrated is a first format, the image format of the first processed image is a second format, the first format is different from the second format, and the image to be calibrated is acquired by a first sensor; A second processing unit, configured to map the first processed image from the first color space of the first sensor to the second color space of a second sensor to determine a first mapped image; A third processing unit, configured to perform color space mapping on the first mapped image to determine a second mapped image; A fourth processing unit, configured to determine a white balance calibration coefficient based on the second mapped image, so as to perform white balance calibration on the image to be calibrated.

8. A communication device, characterized in that, The apparatus includes a processor and a memory. Wherein, a computer program is stored in the memory, and the processor executes the computer program stored in the memory, so that the apparatus executes: the method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A chip, characterized in that, It includes at least one processor and a communication interface; the communication interface is configured to receive signals input into the chip or signals output from the chip, and the processor communicates with the communication interface and implements the method according to any one of claims 1-6 through logic circuits or by executing code instructions.

Images