Image white balance processing method and device, imaging module, equipment, and medium

Abstract

The disclosed embodiments are about an image white balance processing method and device, an image module, an electronic device, and a storage medium, and relate to the field of image technology. The image white balance processing method includes: obtaining an image to be processed, and dividing the image to be processed into multiple image blocks; determining the global white balance coefficient of the image to be processed, and determining the local white balance coefficient of each image block; obtaining a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient; performing white balance processing on the image to be processed according to the target coefficient matrix, and generating a target image corresponding to the image to be processed. The technical solution in the disclosed embodiments can improve the accuracy of white balance.

Description

Technical Field

[0001] The present disclosure relates to the field of imaging technology, and in particular to an image white balance processing method and device, an imaging module, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the process of image white balance processing, image white balance processing is a common way to improve image quality. Image white balance processing can be implemented by AWB (Auto White Balance, automatic white balance) algorithm.

[0003] In the related art, white balance can be performed based on the entire image, in which only the field of view corresponding to a certain light source can be corrected, which has certain limitations. In addition, white balance can be achieved by splicing the white balance results of each image block, but this may result in an uneven transition between image blocks, resulting in a poor white balance effect and reduced image quality.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide an image white balance processing method and device, an image module, an electronic device, and a storage medium, thereby overcoming the problem of poor white balance effect caused by the limitations and defects of related technologies at least to a certain extent.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a method for image white balance processing is provided, comprising: obtaining an image to be processed, and dividing the image to be processed into multiple image blocks; determining a global white balance coefficient of the image to be processed, and determining a local white balance coefficient of each of the image blocks; obtaining a target coefficient matrix based on the global white balance coefficient and the local white balance coefficient; performing white balance processing on the image to be processed according to the target coefficient matrix, and generating a target image corresponding to the image to be processed.

[0008] According to a second aspect of the present disclosure, there is provided an image white balance processing device, comprising: an image segmentation module, used to acquire an image to be processed, and segment the image to be processed to obtain a plurality of image blocks; a coefficient acquisition module, used to determine a global white balance coefficient of the image to be processed, and determine a local white balance coefficient of each of the image blocks; an interpolation module, used to acquire a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient; and an image generation module, used to perform white balance processing on the image to be processed according to the target coefficient matrix, and generate a target image corresponding to the image to be processed.

[0009] According to a third aspect of the present disclosure, an imaging module is provided, comprising: a camera module; a color temperature sensor array, arranged on one side of the camera module, for determining a global white balance coefficient of an image to be processed and a local white balance coefficient of an image block in the image to be processed; an interpolation component, for obtaining a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient; and an image generation component, for performing white balance processing on the image to be processed according to the target coefficient matrix, and generating a target image corresponding to the image to be processed.

[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: an image module as described in any one of the above; a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image white balance processing method of the first aspect and its possible implementation method by executing the executable instructions.

[0011] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image white balance processing method of the first aspect and its possible implementation methods are implemented.

[0012] In the technical solution provided in the embodiment of the present disclosure, on the one hand, since the global white balance coefficient and the local white balance coefficient corresponding to the image block can be determined, and then the target coefficient matrix for white balance processing is determined according to the global white balance coefficient and the local white balance coefficient, so as to perform white balance processing on the image to be processed, when there are multiple light sources in the field of view, the field of view parts corresponding to different light sources can be corrected respectively, avoiding the limitations caused by the need to take into account multiple light sources, increasing the field of view, improving the comprehensiveness of white balance, and determining the local white balance coefficient of each image block to determine the target coefficient matrix, achieving white balance of the entire image through the image block, and improving the accuracy of white balance. On the other hand, through the two systems of global white balance coefficient and local white balance coefficient, the target coefficient matrix is ​​obtained from different dimensions, and the white balance processing is performed on the corresponding image to be processed according to the white balance coefficient in the target coefficient matrix to generate a target image after white balance, so that a smooth transition between image blocks can be achieved in the automatic white balance process, avoiding the appearance of abrupt areas and unnatural phenomena in the entire image, thereby improving the accuracy and stability of white balance and improving the quality of the target image.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0015] Figure 1 A schematic diagram is shown of an application scenario in which the image white balance processing method of an embodiment of the present disclosure can be applied.

[0016] Figure 2 A schematic diagram schematically illustrates an image white balance processing method according to an embodiment of the present disclosure.

[0017] Figure 3 The flowchart of calculating the global white balance coefficient in the embodiment of the present disclosure is schematically shown.

[0018] Figure 4 A flowchart of the target coefficient matrix in an embodiment of the present disclosure is schematically shown.

[0019] Figure 5 The following is a schematic diagram of the process of partition interpolation in an embodiment of the present disclosure.

[0020] Figure 6A schematic diagram schematically illustrates the area division of an embodiment of the present disclosure.

[0021] Figure 7 The flowchart of performing partition interpolation on each region in an embodiment of the present disclosure is schematically shown.

[0022] Figure 8 A schematic diagram for obtaining a target coefficient matrix according to an embodiment of the present disclosure is schematically shown.

[0023] Fig. 9 The flowchart of white balance processing of an image in an embodiment of the present disclosure is schematically shown.

[0024] Fig.10 The flowchart of acquiring the target image in the embodiment of the present disclosure is schematically shown.

[0025] FIG. 11(A) schematically shows a schematic diagram of global white balance in an embodiment of the present disclosure.

[0026] FIG. 11(B) schematically shows a schematic diagram of block white balance in an embodiment of the present disclosure.

[0027] Fig.12 A block diagram of an image white balance processing device in an embodiment of the present disclosure is schematically shown.

[0028] Fig.13 A block diagram of an image module in an embodiment of the present disclosure is schematically shown.

[0029] Fig.14 A block diagram of a color temperature sensor array in an embodiment of the present disclosure is schematically shown.

[0030] Fig.15 The block diagram of the electronic device in the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0032] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0033] In the related art, the image is divided into blocks and identified according to the existing training model, the identified image blocks are excluded, the data of other image blocks are used as reference for calculating the AWB coefficient, and the image is automatically white-balanced using the AWB coefficient. Alternatively, after the image is divided into blocks, the white points are counted according to the weight corresponding to each image block, and the AWB correction coefficient of the entire image is calculated based on the counted white points. Compared with the traditional AWB algorithm, the white point statistics are more accurate and the AWB effect is more stable. Alternatively, the image is divided into blocks, white-balanced according to the characteristics of the image blocks, and then the white-balanced image blocks are spliced ​​together.

[0034] In the above method, using the data of other image blocks as white balance coefficients and calculating the correction coefficients according to the weights of the image blocks is based on making the same AWB correction for the entire image. The AWB correction coefficient is only related to the light source and the image sensor in the camera. When the image sensors are the same, it is only related to the light source. When there are multiple light sources in the field of view, AWB can only correct the field of view corresponding to a certain light source, or compromise by sacrificing the AWB effect to take into account multiple light sources. The results after AWB are spliced ​​together. Since the field of view of each image block is much smaller than that of the entire image, pure color blocks are likely to appear. When the color temperature sensor is not combined, the use of traditional AWB is ineffective for large areas of pure color. The AWB effect of the invalid pure color image block is quite different from the normal AWB effect of the surrounding area, and the performance on the entire image is very abrupt, which will cause an unsmooth and unnatural transition between image blocks.

[0035] In order to solve the technical problems in the related art, an image white balance processing method is provided in an embodiment of the present disclosure, which can be applied to the application scenario of performing white balance processing on an image during the photo-taking process. Figure 1 A schematic diagram showing a system architecture to which the image white balance processing method and device according to the embodiments of the present disclosure can be applied.

[0036] like Figure 1As shown, the terminal 101 may be an intelligent device with an image processing function, such as a smart phone, a computer, a tablet computer, an intelligent speaker, a smart watch, a vehicle-mounted device, a wearable device, a monitoring device, and the like. The terminal may include a camera, and the type of the camera may be any type, as long as it can perform photo processing. The number of cameras may be at least one, such as one, four, etc., as long as it can perform photo processing. The image to be processed may be a captured image, or may be each frame of a captured video.

[0037] In the disclosed embodiment, the terminal 101 may include a memory 102 and a processor 103. The memory is used to store images, and the processor is used to process images, such as performing white balance processing and the like. The memory 102 may store an image 104 to be processed. The terminal 101 obtains the image 104 to be processed from the memory 102, and sends it to the processor 103. The processor 103 performs block processing on the image to be processed to obtain multiple image blocks, and determines the global white balance coefficient of the image to be processed and the local white balance coefficient of each image block. According to the global white balance coefficient and the local white balance coefficient, a target coefficient matrix of the image size is obtained, and further white balance processing is performed on the image to be processed according to the white balance coefficient in the target coefficient matrix, thereby generating a target image 105 after white balance.

[0038] It should be noted that the image white balance processing method provided in the embodiment of the present disclosure may be executed by the terminal 101. The image white balance processing method may also be set in the terminal.

[0039] Next, refer to Figure 2 The image white balance processing method in the embodiment of the present disclosure is described in detail.

[0040] In step S210, an image to be processed is obtained, and the image to be processed is divided into blocks to obtain a plurality of image blocks.

[0041] In the disclosed embodiment, the image to be processed may be an image obtained by photographing the object to be photographed through the camera module of the terminal, or may be each frame of the image in the captured video. The terminal may be any one of the cameras of a smart phone, a digital camera, a smart watch, a wearable device, a vehicle-mounted device, or a monitoring device, as long as it can take a picture of the object to be photographed and can realize image processing. Here, a smart phone is taken as an example for explanation. The camera module may include at least one camera, for example, it may include any one of a main camera, a telephoto camera, a wide-angle camera, a macro camera, or a combination thereof. The image to be processed may be various types of images, for example, it may be a dynamic image or a static image, etc.

[0042] The image to be processed may be an RGB image, that is, an RGB three-channel image. Each pixel of the RGB image is composed of three RGB colors. When the terminal is in a photo mode, the captured image obtained by the camera module may be a RAW image. The RAW image is the original image data information collected by the camera module. In the disclosed embodiment, the captured image captured by the terminal may be converted to obtain an image to be processed. For example, the captured image in RAW format may be converted into an RGB format using a general conversion algorithm to improve the convenience of subsequent processing.

[0043] After the image to be processed is acquired, the image to be processed can be processed in blocks to obtain multiple image blocks. The image block can be a part of the image to be processed, and the multiple image blocks do not overlap. The size of each image block can be the same, and the number of image blocks can be determined according to the block division method. The block division method can be determined according to the sensor state of the terminal, and the corresponding block division method is different for different sensor states. The sensor state can be whether the terminal includes a color temperature sensor array. The color temperature sensor array can be a multi-window color temperature sensor array, and the multi-window color temperature sensor array can be composed of a plurality of color temperature sensor arrays arranged. The color temperature sensor can detect the color temperature of the surrounding environment during the photo shooting process to adjust the color of the image obtained by shooting and improve the accuracy and authenticity of the image. The color temperature sensor array can be set on one side of the camera module and close to the camera module, for example, it can be the left side of the telephoto camera, or the right side of the main camera, or the lower side of the last camera in at least one camera, etc. The specific position of the color temperature sensor array can be determined according to the calibration result of the color temperature sensor array in the actual application process.

[0044] The sensor state may be that the terminal includes a color temperature sensor array or does not include a color temperature sensor array. Based on this, determining a block division method according to the sensor state of the terminal, and then dividing the image to be processed into multiple image blocks according to the block division method may include the following two methods:

[0045] Mode 1: When the sensor state is a color temperature sensor array-containing mode, the image to be processed can be processed in blocks according to the size of the color temperature sensor array contained in the terminal. The size of the color temperature sensor array can be used to represent the number of color temperature sensors contained in the color temperature sensor array. Therefore, the number of image blocks can be determined according to the number of color temperature sensors contained in the color temperature sensor array.

[0046] In some embodiments, the number of color temperature sensors included in the color temperature sensor array may be the same as the number of image blocks, and each color temperature sensor may correspond to an image block. The color temperature, spectrum and other attribute parameters of each image block may be obtained through the color temperature sensor corresponding to each image block, and the white balance coefficient of the image block, i.e., the local white balance coefficient, may be more accurately predicted using the attribute parameters such as color temperature and spectrum. Moreover, the multi-window color temperature sensor array completely matches the field of view of the image sensor included in the camera module of the terminal. The image sensors of the terminal include, but are not limited to, image sensors of camera modules such as the main camera, wide-angle camera, and telephoto camera.

[0047] For example, if there is an M×N multi-window color temperature sensor array in the camera system, the number of image blocks obtained by image segmentation may be the same as the number of color temperature sensors included in the multi-window color temperature sensor array, for example, may also be M×N.

[0048] Since the number of image blocks is the same as the number of color temperature sensors, before determining the number of image blocks, the size of the color temperature sensor array, that is, the number of color temperature sensors included in the color temperature sensor array, can be determined first. Exemplarily, the size of the color temperature sensor array can be determined based on multiple reference attribute parameters. The multiple reference attribute parameters can include, but are not limited to, any one or a combination of field of view characteristics, image resolution, light source status, algorithm performance, and evaluation parameters. Among them, the field of view characteristics can be field of view and light source characteristics, the light source status can be the number and distribution of light sources, the algorithm performance can be the amount of calculation, and the evaluation parameter can be white balance accuracy.

[0049] The process of determining the color temperature sensor array by using multiple reference attribute parameters may include: first, since the field of view of each color temperature sensor is a circular area, and the light source diffuses to all sides at the same time, the square effect is better than the rectangle for the shape of the image block corresponding to each field of view, so the shape of the image block can be square. Secondly, if the resolution of all cameras (main camera, wide angle, telephoto) of the terminal is 4096×3072, and the image aspect ratio is 4:3, when the image to be processed can be divided and evenly divided into square image blocks, the number of image blocks can be any one of 4×3, 8×6, and 12×9. Thirdly, when the distance between the light source and the photographed object is close, the total number of light sources in the field of view of a shooting scene is generally less than 10, and the light sources distributed in the same image block are generally not more than 2 light sources. The number of image blocks obtained after image segmentation is greater than 10 to meet the requirements, so the number of image blocks can be any one of 4×3, 8×6 or 12×9. Furthermore, since each image block needs to calculate white balance coefficients, CCM (Color Correction Matrix) parameters, and filtering operations, the larger the number of image blocks, the greater the amount of calculation, so the fewer the number of divided image blocks, the better. Based on this, the number of image blocks can be, for example, 4×3 or 8×6. Finally, since the automatic white balance effect is related to the image content of the image to be processed, the larger the field of view, the richer the content, and the higher the accuracy of the automatic white balance. When the field of view of the entire image is fixed, the larger the size of the image block obtained by image segmentation, the larger the field of view corresponding to the image block; when the image segmentation is small, pure color image blocks may appear, resulting in color unevenness and color distortion, etc., and the stability of the automatic white balance of the pure color image block is poor, so the larger the size of the divided image block, the better, and the fewer the number of image blocks, the better, so the number of image blocks can be 4×3 or 8×6. Based on this, the size of the color temperature sensor array can be 4×3 or 8×6. Correspondingly, the number of image blocks is consistent with the size of the color temperature sensor array, so that each image sensor can correspond to an image block respectively. The number of image blocks may be, for example, 4×3 or 8×6.

[0050] In the embodiment of the present disclosure, the number of image blocks is determined according to the size of the color temperature sensor array, so that the image to be processed can be reasonably divided into blocks, thereby improving the accuracy of image division.

[0051] Method 2: When the sensor state is a type that does not include a color temperature sensor array, the number of image blocks can be determined based on multiple reference attribute parameters. The multiple reference attribute parameters may include, but are not limited to, any one or a combination of field of view characteristics, image resolution, light source status, algorithm performance, and evaluation parameters. Among them, the field of view characteristics may be field of view and light source characteristics, the light source status may be the number and distribution of light sources, the algorithm performance may be the amount of calculation, and the evaluation parameter may be white balance accuracy. Based on this, the number of image blocks obtained by image division can be determined based on actual needs on the basis of the reference attribute parameters. The specific method for determining the number of image blocks based on multiple reference attribute parameters may be the same as the specific method in method 1, and will not be repeated here. It should be noted that the number of image blocks is determined based on multiple reference attribute parameters, but each image block cannot correspond to the color temperature sensor included in the color temperature sensor array.

[0052] Next, continue to refer to Figure 2 As shown in , in step S220, the global white balance coefficient of the image to be processed is determined, and the local white balance coefficient of each image block is determined.

[0053] In the disclosed embodiment, the global white balance coefficient refers to the white balance coefficient used to perform automatic white balance processing on the entire image to be processed. The local white balance coefficient refers to the white balance coefficient used to perform white balance on a portion of the image represented by an image block. When the image to be processed is divided into blocks and the image blocks are used to perform automatic white balance processing, the accuracy of the white balance coefficient calculated when the color content of some image blocks is not rich or there are pure color image blocks is relatively poor, so that the result obtained after the automatic white balance correction is inaccurate, and there will be a large difference between the effects of the automatic white balance correction between these image blocks and other image blocks in the image to be processed. In order to reduce the difference between the image blocks, the local white balance coefficient of each image block can be corrected. Exemplarily, the local white balance coefficients of all image blocks can be corrected so that the local white balance coefficients of all image blocks float within a specified range above and below a preset white balance correction level. Among them, the preset white balance correction level can be represented by a global white balance coefficient. Therefore, the local white balance coefficient can be corrected according to the global white balance coefficient so that the local white balance coefficient of the image block is within a preset floating range.

[0054] When determining the global white balance coefficient, it can be determined in combination with the sensor status. Figure 3 A flowchart for determining the global white balance coefficient is schematically shown in FIG. Figure 3 As shown in , it mainly includes the following steps:

[0055] In step S310, it is determined whether there is a color temperature sensor array in the terminal; if so, the process goes to step S320 or step S330; if not, the process goes to step S340.

[0056] In step S320, a global attribute parameter is obtained according to the attribute parameter of each image block obtained by the color temperature sensor array, and the global attribute parameter is processed to obtain the global white balance coefficient.

[0057] In this step, when the sensor state is that the terminal includes a color temperature sensor array, each color temperature sensor in the color temperature sensor array can obtain the attribute parameters of the image block corresponding to it. The attribute parameter may be a spectrum or a color temperature. When the attribute parameter is a spectrum, the global attribute parameter may be a global spectrum. The spectrum is used to represent various color distributions. Based on this, a global spectrum may be obtained according to the spectrum of each image block obtained by the color temperature sensor array, and the global spectrum may be processed to obtain the global white balance coefficient. Exemplarily, the spectra of each image block may be averaged to obtain a global spectrum, and then the global spectrum may be input into a trained recognition model to predict and identify the corresponding global white balance coefficient. The recognition model may be any suitable machine learning model or other model, as long as the global white balance coefficient can be determined. For example, the recognition model may be a neural network model.

[0058] In addition, when the attribute parameter is color temperature, the global attribute parameter may be a global color temperature. Based on this, a global color temperature value may be obtained according to the color temperature of each image block obtained by the color temperature sensor array, and the global color temperature value may be interpolated to obtain the global white balance coefficient. Exemplarily, all color temperatures corresponding to all image blocks output by the color temperature sensor array may be averaged to obtain a global color temperature value, and a white balance coefficient sequence corresponding to the color temperature value sequence may be obtained, and linear interpolation may be performed in the color temperature value sequence according to the global color temperature value, thereby obtaining a global white balance coefficient. Among them, the color temperature value sequence may be a sequence composed of color temperature values ​​of multiple light sources, and the white balance coefficient sequence corresponding to the color temperature value sequence may be obtained by using the automatic white balance light box tool to mark the corresponding white balance coefficient for each light source and record the color temperature value of the light source. For example, the white balance coefficient of each light source of each color temperature may be marked by the light box tool, and then the white balance coefficients of multiple color temperatures may be arranged in order to form a white balance coefficient sequence. The global color temperature value is further linearly interpolated in the color temperature value sequence to obtain an interpolation result, and the interpolation result and the white balance coefficient sequence are combined to obtain the corresponding global white balance coefficient.

[0059] In step S330, the local white balance coefficients of all image blocks are averaged to obtain the global white balance coefficient.

[0060] In this step, first, the local white balance coefficient of each image block can be calculated according to the attribute parameters output by the color temperature sensor array. The specific method is the same as that in step S320 and will not be repeated here. Furthermore, the local white balance coefficients of all image blocks can be averaged to obtain the global white balance coefficient of the image to be processed. For example, the local white balance coefficient 1 of image block 1, the local white balance coefficient 2 of image block 2, ... and the local white balance coefficient N of image block N are added, and the addition result is divided by the number of image blocks N to obtain the ratio between the two, so as to determine the global white balance coefficient of the image to be processed according to the ratio. By averaging the local white balance coefficients of all image blocks, the efficiency and speed of obtaining the global white balance coefficient can be improved.

[0061] In step S340, the global white balance coefficients are determined using the candidate calculation method.

[0062] In this step, when the sensor state is that the terminal does not include a color temperature sensor array, since the color temperature sensor array does not exist, the global white balance coefficient cannot be calculated according to the attribute parameters output by the color temperature sensor array. In this case, the global white balance coefficient of the image to be processed can be calculated according to a candidate calculation method. The candidate calculation method may include but is not limited to a gray world method, a total reflection method (white block method), a white point-color temperature calibration method, automatic white balance processing by coordinate data of image blocks of a marked preset color, or calculating the white balance coefficient according to the white point of weighted statistics corresponding to each image block.

[0063] Among them, the specific steps of the gray world method may include: calculating the average value of each color channel in the image to be processed; determining a reference value K, for example, Gmean can be selected; calculating the gain value corresponding to each pixel through Rgain = Gmean / Rmean, Bgain = Gmean / Bmean; multiplying each pixel in the image to be processed by the corresponding gain value for correction. The specific steps of the total reflection method may include: searching for the brightest point in the image to be processed with the largest RGB value as the target point; calculating the Rgain and Bgain of the target point through Rgain = G / R, Bgain = G / B; applying the calculated Rgain and Bgain to each pixel for correction. The specific steps of the white point-color temperature calibration method may include: obtaining the R / G and B / G values ​​of white imaging of different sensors under different color temperature light sources measured in a laboratory environment; obtaining the R / G and B / G values ​​of the entire image to be processed in a real environment to determine the current color temperature; determining the corresponding Rgain and Bgain according to the current color temperature; applying the calculated Rgain and Bgain to each pixel for correction. In addition, the image can be divided into blocks for identification according to the existing training model, excluding the identified image blocks, using the data of other image blocks as a reference for calculating the white balance coefficient, and automatically white balancing the image using the white balance coefficient. It is also possible to divide the image into blocks, count the white points according to the weights corresponding to each image block, and calculate the white balance correction coefficient of the entire image based on the white points after counting.

[0064] In the disclosed embodiment, the global white balance coefficient of the image to be processed is calculated in an appropriate manner according to the sensor state, thereby avoiding the problem that one algorithm is used in all states and cannot be used in some scenes, and improving the accuracy of the global white balance coefficient. In addition, if the sensor state is that there is a color temperature sensor array, the global white balance coefficient can be determined in combination with the color temperature sensor array, thereby improving accuracy, stability and white balance effect.

[0065] When determining the local white balance coefficient, it can also be determined in combination with the sensor state. Exemplarily, when the sensor state is that a color temperature sensor array is included, each image block corresponds to a color temperature sensor in the color temperature sensor array, and local attribute parameters can be obtained based on the attribute parameters of each image block obtained by the color temperature sensor array, and the local attribute parameters are processed to obtain the local white balance coefficient. The attribute parameters can be spectrum and color temperature. In some embodiments, the spectrum output by the color temperature sensor corresponding to each image block can be input into a trained recognition model to determine the corresponding local white balance coefficient. It can also be calculated by using the color temperature value interpolation light box light source color temperature method. Specifically, all color temperatures corresponding to each image block output by the color temperature sensor array are determined as local color temperature values, and a white balance coefficient sequence corresponding to the color temperature value sequence obtained for each light box light source is obtained, and linear interpolation is performed in the color temperature value sequence according to the local color temperature value to obtain the local white balance coefficient. If the sensor state is that the terminal does not include a color temperature sensor array, the local white balance coefficient of the image block can be calculated according to a candidate calculation method. The specific method for calculating the local white balance coefficient of the image block is as follows. Figure 3 Step S310, step S320 and step S340 are the same as those in the above description and will not be repeated here.

[0066] In some embodiments, the global white balance coefficient may be the ratio of the mean of R and B to the mean of G of all gray points in the image to be processed, that is, R mean / G mean and B mean / G mean. The local white balance coefficient may be the R / G value and B / G value of each image block.

[0067] It should be noted that the global white balance coefficient of the image to be processed and the local white balance coefficient of each image block can be calculated simultaneously, or the global white balance coefficient can be calculated first and then the local white balance coefficient, or the local white balance coefficient can be calculated first and then the global white balance coefficient. There is no special limitation here.

[0068] In the embodiment of the present disclosure, since one color temperature sensor in the color temperature sensor array can correspond to one image block, the color temperature sensor array can obtain the attribute parameters of each image block. Therefore, the attribute parameters of each image block obtained by the color temperature sensor can more accurately determine the local white balance coefficient and the global white balance coefficient of each image block, thereby improving the accuracy of the white balance coefficient.

[0069] Next, continue to refer to Figure 2 As shown in , in step S230, a target coefficient matrix is ​​obtained according to the global white balance coefficient and the local white balance coefficient.

[0070] In the disclosed embodiment, in the process of performing white balance through image blocks, the accuracy of the calculated white balance coefficient is low due to the fact that some image blocks are not rich in color content, or there are pure color image blocks, and the difference between the image blocks after correction is large. In order to reduce the difference between the image blocks, the target coefficient matrix can be obtained through the global white balance coefficient and the local white balance coefficient of each image block. Exemplarily, the target coefficient matrix can be obtained by adjusting the local white balance coefficient of each image block through the global white balance coefficient; the target coefficient matrix can also be obtained by interpolating the global white balance coefficient through the local white balance coefficient; and the global white balance coefficient and the local white balance coefficient can also be fused according to the corresponding weights to obtain the target coefficient matrix. Here, the example of adjusting the local white balance coefficient through the global white balance coefficient to obtain the target coefficient matrix is ​​used for explanation.

[0071] Exemplarily, the target coefficient matrix can be interpolated based on the global white balance coefficient and the local white balance coefficient. The target coefficient matrix is ​​used to perform white balance processing on the image to be processed. In some embodiments, the target coefficient matrix can represent the white balance coefficient corresponding to each pixel in the image to be processed, or the white balance coefficient corresponding to some pixels, or the global white balance coefficient corresponding to the entire image to be processed. The range represented by the target coefficient matrix is ​​determined according to actual needs. Here, the target coefficient matrix is ​​used as an example to represent the white balance coefficient corresponding to each pixel in the image to be processed.

[0072] Each pixel in the image to be processed may include a first color sub-pixel R, a second color sub-pixel B, and a third color sub-pixel G. In some embodiments, the numerical range of the local white balance coefficient may be limited by the global white balance coefficient to obtain a limited local white balance coefficient. After determining the limited local white balance coefficient, the limited local white balance coefficient may be interpolated to adjust the local white balance coefficient by the global white balance coefficient, thereby interpolating and determining a target coefficient matrix to interpolate the white balance coefficient from the color temperature sensor array size to the target coefficient matrix of the image to be processed size. For the target coefficient matrix obtained after interpolation, each pixel has a corresponding white balance coefficient, which improves comprehensiveness.

[0073] Figure 4 A flowchart for obtaining the target coefficient matrix is ​​schematically shown in FIG. Figure 4 As shown in , it mainly includes the following steps:

[0074] In step S410, the value range of the local white balance coefficient is limited by the global white balance coefficient to obtain a limited local white balance coefficient.

[0075] In the embodiment of the present disclosure, the global white balance coefficient can be used as a reference to adjust the numerical range of the local white balance coefficient within a preset floating range to determine the limited local white balance coefficient. The preset floating range can be set according to actual needs, for example, it can be 0.2 or any suitable value. Based on this, the global white balance coefficient can be used as a reference to control the local white balance coefficient to float within the preset floating range above and below it, so as to obtain the limited local white balance coefficient. When the preset floating range is 0.2, the local white balance coefficient of each image block is within the range of 0.8 times to 1.2 times the global white balance coefficient. When it is less than 0.8 times, it is determined as 0.8 times the global white balance coefficient, and when it is greater than 1.2 times, it is determined as 1.2 times the global white balance coefficient. For example, when the local white balance coefficient of a certain image block is 0.6 times the global white balance coefficient, it is determined as 0.8 times the global white balance coefficient; when the local white balance coefficient of a certain image block is 2 times the global white balance coefficient, it is determined as 1.2 times the global white balance coefficient. By limiting the numerical range of the local white balance coefficient within a preset floating range through the global white balance coefficient, the problem of large differences between the global white balance coefficient and the local white balance coefficient can be reduced, and the consistency of the global white balance coefficient and the local white balance coefficient can be improved.

[0076] In step S420, the restricted local white balance coefficients are interpolated to determine the target coefficient matrix.

[0077] In the disclosed embodiment, when obtaining the restricted local white balance coefficient, since the restricted local white balance coefficient is still the local white balance coefficient corresponding to the image block, in order to achieve white balance processing of the entire image to be processed, the restricted local white balance coefficient corresponding to the image block can be interpolated into a white balance coefficient matrix of the image size of the image to be processed. The white balance coefficient matrix can be represented by a target coefficient matrix, that is, the local white balance coefficient of each image block is interpolated from the color temperature sensor array size to the white balance coefficient matrix of the image size of the image to be processed, that is, the target coefficient matrix.

[0078] It should be added that, in order to eliminate the abrupt and unnatural phenomenon of the white balance effect between image blocks, before the local white balance coefficient of the image block is interpolated, the limited local white balance coefficient of the image block can be smoothed. The smoothing method may include any one of mean filtering, Gaussian filtering, median filtering, maximum and minimum filtering, or a combination thereof. In some embodiments, the limited local balance coefficient of the image block can be smoothed by first median filtering and then mean filtering. Exemplarily, the limited local white balance coefficient can be subjected to median filtering to obtain a median filtering result, and the median filtering result can be subjected to mean filtering for smoothing. Median filtering refers to setting the gray value of each pixel point to the median value of all pixel points in a certain neighborhood window of the point. Mean filtering refers to replacing the pixel point with the mean of the neighborhood of the pixel point. By combining median filtering and mean filtering, the difference between the local white balance coefficients of the image blocks can be reduced, the unnatural effect of the white balance effect between the image blocks can be avoided, and the accuracy of the local white balance coefficient can be improved.

[0079] In the disclosed embodiment, when the restricted local white balance coefficients are interpolated, the restricted local white balance coefficients of multiple image blocks can be interpolated by partition to obtain a target coefficient matrix. Partition interpolation means that different regions are interpolated by using corresponding interpolation methods. Different regions can have different interpolation methods.

[0080] Figure 5 The schematic diagram of the process of partition interpolation is shown in FIG. Figure 5 As shown in , it mainly includes the following steps:

[0081] In step S510 , a first area, a second area, and a third area are determined.

[0082] In this step, multiple regions may be determined in the image to be processed, and the multiple regions may include a first region, a second region, and a third region. The first region, the second region, and the third region constitute the entire region of the image to be processed, and do not overlap with each other. The first region, the second region, and the third region may include a sub-image block in at least one image block, for example, may include a sub-image block in an image block, or may include multiple sub-image blocks in multiple image blocks.

[0083] refer to Figure 6The schematic diagram of the area division shown in , the image to be processed can be divided into 4×5 image blocks. For example, image block 601 is an image block in which the image to be processed 600 is divided according to the size of the color temperature sensor array. Each image block can include multiple sub-image blocks 602, for example, four sub-image blocks. Exemplarily, each image block can be divided into four equal parts to obtain multiple sub-image blocks. On this basis, the sub-image blocks located at the corners of the image to be processed form the first area, the sub-image blocks located at the edges of the image to be processed form the second area, and the sub-image blocks located in the middle of the image to be processed form the third area. Reference Figure 6 As shown in the figure, the first area 603 is the area formed by the four sub-image blocks at the four corners of the image to be processed; the second area 604 is the area formed by the sub-image blocks at the four sides of the image to be processed; the third area 605 is the area formed by the sub-image blocks in the middle of the image to be processed, that is, the remaining middle area.

[0084] In step S520, different interpolation methods are used to interpolate the first region, the second region, and the third region to determine the target coefficient matrix.

[0085] In this step, after a plurality of regions are determined, different interpolation methods may be used to interpolate the local white balance coefficients of the sub-image blocks in each region, thereby obtaining a target coefficient matrix. Figure 7 A flowchart for performing interpolation is schematically shown in FIG. Figure 7 The steps in belong to the specific implementation of step S520. Figure 7 As shown in , it mainly includes the following steps:

[0086] In step S710, based on the restricted local white balance coefficient corresponding to the image block to which the sub-image block of the first region belongs, an interpolation coefficient corresponding to the sub-image block of the first region is determined;

[0087] In step S720, linear interpolation is performed on restricted local white balance coefficients corresponding to image blocks to which adjacent sub-image blocks in the second region belong, to determine interpolation coefficients corresponding to sub-image blocks in the second region;

[0088] In step S730, bilinear interpolation is performed on restricted local white balance coefficients corresponding to image blocks to which adjacent sub-image blocks in the third region belong, and interpolation coefficients corresponding to sub-image blocks in the third region are determined to determine the target coefficient matrix.

[0089] In the disclosed embodiment, for each image block, the value of the center point of the image block can be used as the local white balance coefficient of the image block. The value of the center point can be, for example, the white balance coefficient of the center point of the image block. When interpolating sub-image blocks in different regions, interpolation can be performed based on the values ​​of the center points of the image blocks corresponding to the sub-image blocks in different regions. It should be noted that the value interpolated here can be the restricted local white balance coefficient of the center point.

[0090] For the sub-image block of the first region, the restricted local white balance coefficient of the image block to which the sub-image block belongs can be used as the interpolation coefficient corresponding to the sub-image block. The interpolation coefficient corresponding to the sub-image block of the first region is used to describe the interpolation result obtained by interpolating each sub-image block of the first region, and specifically can be the interpolation coefficient of the sub-image block of the first region.

[0091] For the sub-image blocks in the second region, linear interpolation is performed on the restricted local white balance coefficients corresponding to the image blocks to which the adjacent sub-image blocks belong, to obtain the interpolation coefficients corresponding to the sub-image blocks in the second region. The adjacent sub-image blocks here may be two adjacent sub-image blocks, for example, two adjacent sub-image blocks in the X direction, or two adjacent sub-image blocks in the Y direction. Linear interpolation refers to predicting a white balance coefficient in one direction as an interpolation coefficient based on the restricted local white balance coefficients of the image blocks to which the two adjacent sub-image blocks belong, for example, predicting in the X direction and the Y direction respectively, until linear interpolation is performed for all sub-image blocks, thereby predicting a white balance coefficient in the X direction and the Y direction. The interpolation coefficients corresponding to the sub-image blocks in the second region can be used to describe the interpolation results obtained by linear interpolation of every two adjacent sub-image blocks in the second region, and specifically can be the interpolation coefficients generated based on the sub-image blocks in the second region.

[0092] For the sub-image blocks in the third region, bilinear interpolation is performed on the restricted local white balance coefficients corresponding to the image blocks to which the adjacent sub-image blocks belong, to obtain the interpolation coefficients corresponding to the sub-image blocks in the third region. The adjacent sub-image blocks here can be four adjacent sub-image blocks, or can be all the sub-image blocks in the third region. Four adjacent sub-image blocks are taken as an example for explanation here. Bilinear interpolation refers to predicting a white balance coefficient in two directions respectively according to the restricted local white balance coefficients corresponding to the image blocks to which the four adjacent sub-image blocks belong, until all the sub-image blocks in the third region are interpolated, thereby determining the interpolation coefficients corresponding to the sub-image blocks in the third region. The interpolation coefficients corresponding to the sub-image blocks in the third region can be used to describe the interpolation results obtained by linearly interpolating every two adjacent sub-image blocks in the third region, and specifically can be the interpolation coefficients generated according to the sub-image blocks in the third region.

[0093] refer to Figure 8As shown in , each image block may include four sub-image blocks, and the sizes of the four sub-image blocks may be the same. Sub-image block 8001 in the first area belongs to image block 811. Sub-image block 8002 in the second area belongs to image block 811, and sub-image block 8003 in the second area belongs to image block 812; sub-image block 8008 in the second area belongs to image block 811, and sub-image block 8009 in the second area belongs to image block 813. Sub-image block 8004 in the third area belongs to image block 811, and sub-image block 8005 in the third area belongs to image block 812; sub-image block 8006 in the third area belongs to image block 813; and sub-image block 8007 in the third area belongs to image block 814. Based on this, for the sub-image blocks in the first area, the restricted local white balance coefficient represented by the value of the center point of image block 811 can be directly determined as the interpolation coefficient. For the sub-image blocks in the second region, the restricted local white balance coefficients of the image blocks 811 and 812 are linearly interpolated in the X direction to obtain the interpolation coefficients, that is, the linear interpolation is performed according to the values ​​of the center points of the image blocks 811 and 812. The restricted local white balance coefficients of the image blocks 811 and 813 can also be linearly interpolated in the Y direction to obtain the interpolation coefficients, thereby obtaining the interpolation coefficients of the second region. For the sub-image blocks in the third region, the local white balance coefficients of the image blocks 811, 812, 813, and 814 are bilinearly interpolated in the X and Y directions to obtain the interpolation coefficients, that is, the values ​​of the center points of the image blocks 811, 812, 813, and 814 are bilinearly interpolated.

[0094] Repeat the above steps S710 to S730 until all sub-image blocks in all regions of the multiple image blocks are partitioned and interpolated to obtain corresponding interpolation coefficients, so as to generate a target coefficient matrix based on the obtained interpolation coefficients.

[0095] In the disclosed embodiment, by performing partition interpolation on sub-image blocks in different areas, the white balance coefficients of the image blocks are interpolated from the color temperature sensor array size to a target coefficient matrix represented by a white balance coefficient matrix of the image size of the image to be processed, so that each pixel in the image to be processed corresponds to a white balance coefficient, which can improve the accuracy and comprehensiveness of the white balance coefficient.

[0096] Next, continue to refer to Figure 2 As shown in , in step S240, white balance processing is performed on the image to be processed according to the target coefficient matrix to generate a target image corresponding to the image to be processed.

[0097] In the disclosed embodiment, when white balance processing is performed on the image to be processed according to the target coefficient matrix, the parameters of each pixel in the image to be processed can be corrected to achieve white balance processing. Each pixel refers to a pixel composed of RGB in the image to be processed, and each pixel may include a first color sub-pixel R, a second color sub-pixel B, and a third color sub-pixel G. The parameter refers to the color parameter of the sub-pixel to be adjusted, and specifically, it can be the channel component of the color channel represented by the sub-pixel to be adjusted. The automatic white balance algorithm is used to identify and find the gray points in the image to be processed, and the R and B mean values ​​of all gray points are found and compared with the G mean value, and the ratio is used to correct the R and B values ​​of all pixels in the image. Therefore, the sub-pixel to be adjusted can be the first color sub-pixel and the second color sub-pixel. On this basis, the first color sub-pixel can be, for example, R or B, and the second color sub-pixel is different from the first color sub-pixel, for example, it can be B or R.

[0098] Since both the global white balance coefficient and the local white balance coefficient may include R / G values ​​and B / G values, the target coefficient matrix may include a first matrix and a second matrix. In combination with the composition of each pixel, the first matrix may be a white balance coefficient matrix of a first color sub-pixel, and the second matrix may be a white balance coefficient matrix of a second color sub-pixel. Exemplarily, the first matrix may be an Rgain matrix, and the second matrix may be a Bgain matrix. The first matrix and the second matrix are respectively used to adjust and correct the parameters of the corresponding color sub-pixels, thereby adjusting the color and color temperature of the image to be processed.

[0099] On the basis of the automatic white balance algorithm, when adjusting the parameters of the pixel points of the image to be processed, the parameters of the first color sub-pixel and the second color sub-pixel in the image to be processed can be corrected according to the target coefficient matrix to obtain the target image. Specifically, the following steps can be included: correcting the parameters of the first color sub-pixel according to the first matrix in the target coefficient matrix; correcting the parameters of the second color sub-pixel according to the second matrix in the target coefficient matrix. The first matrix is ​​the white balance matrix of the first color sub-pixel, and the second matrix is ​​the white balance matrix of the second color sub-pixel.

[0100] In the disclosed embodiment, each pixel in the image to be processed can be automatically corrected for white balance using the corresponding white balance coefficient. Exemplarily, the parameters of the first color sub-pixel in each pixel can be corrected according to the white balance matrix of the first color sub-pixel contained in the target coefficient matrix, and the parameters of the second color sub-pixel in each pixel can be corrected according to the white balance matrix of the second color sub-pixel contained in the target coefficient matrix, so as to achieve parameter correction for each pixel using the corresponding white balance coefficient. In some embodiments, the parameters of the first color sub-pixel in each pixel can be multiplied by the first matrix to perform automatic white balance correction on the parameters of the first color sub-pixel. At the same time, the parameters of the second color sub-pixel in each pixel can be multiplied by the second matrix to perform automatic white balance correction on the parameters of the second color sub-pixel in each pixel in the image to be processed. That is, the R value is multiplied by the first matrix Rgain, and the B value is multiplied by the second matrix Bgain to perform white balance coefficient correction.

[0101] In the disclosed embodiment, since the global white balance coefficient and the local white balance coefficient of each image block can be determined, and the local white balance coefficient is adjusted by the global white balance coefficient, a target coefficient matrix obtained by interpolation of the global white balance coefficient and the local white balance coefficient is obtained, and then the parameters of the first color sub-pixel and the second color sub-pixel of all pixels in the processed image are corrected according to the target coefficient matrix, the accuracy of the pixel parameters is improved, and the accuracy and stability of the white balance are improved, so that the white balance effect is more realistic and reliable.

[0102] After the channel components of the first color sub-pixels and the second color sub-pixels of all pixels are corrected and transformed, a target image corresponding to the image to be processed can be obtained. The target image can be an image obtained by automatically white balancing the image to be processed. The image after automatic white balance correction is RGB three-channel data, that is, an RGB image. In the ISP (Image Signal Processing) flow, the target image can be converted according to actual needs to change the format of the target image. For example, the RGB image is converted into an image in RAW format or YUV format to facilitate image processing by subsequent modules.

[0103] Compared with the global white balance algorithm, the block white balance in the disclosed embodiment can respectively correct the parts of the field of view corresponding to different light sources when there are multiple light sources in the field of view, thus avoiding limitations, increasing the field of view, and improving the accuracy and comprehensiveness of white balance. In addition, the transition between image blocks is smooth and natural, and there will be no abrupt areas on the entire image, which improves smoothness and image quality.

[0104] Fig. 9 A flowchart for performing white balance processing is schematically shown in FIG. Fig. 9 As shown in , it mainly includes the following steps:

[0105] In step S901, an image to be processed is obtained. The image to be processed is an RGB image, which can be obtained by converting a RAW image.

[0106] In step S902, the image to be processed is divided into blocks to obtain a plurality of image blocks. Exemplarily, the image to be processed may be divided into blocks according to a multi-window color temperature sensor array, or the number of image blocks may be preset.

[0107] In step S903, the global white balance coefficient of the image to be processed is determined.

[0108] In step S904, the local white balance coefficient of the image block in the image to be processed is determined.

[0109] In step S905, the local white balance coefficient is limited. For example, the value range of the local white balance coefficient can be limited by the global white balance coefficient.

[0110] In step S906, the local white balance coefficients are smoothed.

[0111] In step S907, the restricted local white balance coefficients are interpolated to obtain a target coefficient matrix. Exemplarily, each image may include multiple sub-image blocks, and the multiple sub-image blocks may be divided into different regions, such as a first region, a second region, and a third region. Different interpolation methods may be used to perform partition interpolation on sub-image blocks located in different regions.

[0112] In step S908, the image to be processed is corrected and transformed for white balance processing. For example, the correction transformation can be performed on the parameters of the first color sub-pixel and the second color sub-pixel in the pixel point of the image to be processed in the target coefficient matrix, and the correction transformation can be realized by multiplying the target coefficient matrix with the first color sub-pixel and the second color sub-pixel.

[0113] In step S909, a target image is output, which may be an image to be processed after automatic white balance.

[0114] Fig.10 A flowchart for acquiring a target image is schematically shown in FIG. Fig.10 As shown in , it mainly includes the following steps:

[0115] In step S1001, an image 1010 to be processed is obtained.

[0116] In step S1002 , the image to be processed is divided into blocks to obtain a plurality of image blocks 1020 .

[0117] In step S1003 , a global white balance coefficient 1040 and a local white balance coefficient 1050 are acquired based on the color temperature sensor array 1030 .

[0118] In step S1004 , a target coefficient matrix 1060 is generated according to the global white balance coefficient 1040 and the local white balance coefficient 1050 .

[0119] In step S1005 , the parameters of the first color sub-pixel and the second color sub-pixel of each pixel in the image to be processed 1010 are corrected according to the target coefficient matrix 1060 .

[0120] In step S1006 , a target image 1070 is generated.

[0121] Figure A in Figure 11 is a global white balance image, and Figure B in Figure 11 is an image block white balance image. In comparison, the image blocks obtained by image block white balance are smoother, and the image quality of the image block white balance image is higher than that of the global white balance image.

[0122] The technical solution in the disclosed embodiment combines a multi-window color temperature sensor array to obtain the global white balance coefficient and the local white balance coefficient of the image block. Compared with the traditional automatic white balance algorithm, the white balance coefficient is calculated for all image blocks separately, which can better handle the white balance problem of multiple light sources with different color temperatures, improve the accuracy and stability of automatic white balance. In addition, the automatic white balance algorithm based on image segmentation, since there is no overlapping area between image blocks, the multi-light source mixed scene can be automatically corrected by image blocks, avoiding the limitation caused by the related technology that only the field of view corresponding to a certain light source can be corrected, and can improve comprehensiveness and accuracy. The sub-image blocks of multiple image blocks are interpolated by partition interpolation, and are enlarged to a target coefficient matrix that matches the image size of the image to be processed, so that they correspond to each pixel in the image to be processed, and then the parameters of each pixel in the image to be processed are adjusted according to the target coefficient matrix. This avoids the problem of pure color blocks appearing due to a small field of view, which leads to the failure of white balance correction for pure color blocks and the large difference in white balance effects between the pure color blocks and the white balance effects of other surrounding image blocks. This can improve the effect and authenticity of block white balance, achieve smooth transition between image blocks, and improve image quality.

[0123] The present disclosure provides an image white balance processing device, referring to Fig.12 As shown in , the image white balance processing device 1200 may include:

[0124] The image segmentation module 1201 is used to obtain an image to be processed and segment the image to be processed into multiple image blocks;

[0125] A coefficient acquisition module 1202, used to determine the global white balance coefficient of the image to be processed, and determine the local white balance coefficient of each image block;

[0126] An interpolation module 1203 is used to obtain a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient;

[0127] The image generation module 1204 is used to perform white balance processing on the image to be processed according to the target coefficient matrix to generate a target image corresponding to the image to be processed.

[0128] In an exemplary embodiment of the present disclosure, the coefficient acquisition module includes: a range limiting module, which is used to limit the numerical range of the local white balance coefficient by using the global white balance coefficient to obtain the restricted local white balance coefficient.

[0129] In an exemplary embodiment of the present disclosure, the range limiting module includes: a coefficient adjustment module, which is used to adjust the local white balance coefficient within a preset floating range based on the global white balance coefficient to determine the limited local white balance coefficient.

[0130] In an exemplary embodiment of the present disclosure, the coefficient acquisition module includes: a coefficient matrix determination module, which is used to interpolate the restricted local white balance coefficients to determine the target coefficient matrix.

[0131] In an exemplary embodiment of the present disclosure, the coefficient matrix determination module includes: a partition interpolation module, which is used to perform partition interpolation on the restricted local white balance coefficients of the multiple image blocks to determine a target coefficient matrix.

[0132] In an exemplary embodiment of the present disclosure, a partition interpolation module includes: a region division module, used to determine a first region, a second region and a third region; wherein the image block includes a plurality of sub-image blocks; the sub-image blocks located at the corners of the image to be processed form the first region, the sub-image blocks located at the edges of the image to be processed form the second region, and the sub-image blocks located in the middle of the image to be processed form the third region; and an interpolation control module, used to interpolate the first region, the second region and the third region using different interpolation methods, respectively, to determine the target coefficient matrix.

[0133] In an exemplary embodiment of the present disclosure, the interpolation control module includes: a first coefficient determination module, which is used to determine the interpolation coefficients corresponding to the sub-image blocks in the first area based on the restricted local white balance coefficients corresponding to the image blocks to which the sub-image blocks in the first area belong; a second coefficient determination module, which is used to perform linear interpolation on the restricted local white balance coefficients corresponding to the image blocks to which the adjacent sub-image blocks in the second area belong, and determine the interpolation coefficients corresponding to the sub-image blocks in the second area; and a third coefficient determination module, which is used to perform bilinear interpolation on the restricted local white balance coefficients corresponding to the image blocks to which the adjacent sub-image blocks in the third area belong, and determine the interpolation coefficients corresponding to the sub-image blocks in the third area, so as to determine the target coefficient matrix.

[0134] In an exemplary embodiment of the present disclosure, before interpolating the restricted local white balance coefficient to determine the target coefficient matrix, the device also includes: a first filtering module, used to perform median filtering on the restricted local white balance coefficient to obtain a median filtering result; a second filtering module, used to perform mean filtering on the median filtering result for smoothing.

[0135] In an exemplary embodiment of the present disclosure, the image generation module includes: a color correction module, which is used to correct the parameters of the first color sub-pixel and the second color sub-pixel of each pixel point in the image to be processed according to the target coefficient matrix to obtain the target image.

[0136] In an exemplary embodiment of the present disclosure, the target coefficient matrix includes a first matrix and a second matrix; the parameter correction module includes: a first color correction module, used to correct the parameters of the first color sub-pixel according to the first matrix in the target coefficient matrix; a second color correction module, used to correct the parameters of the second color sub-pixel according to the second matrix in the target coefficient matrix; wherein the matrix is ​​the white balance coefficient matrix of the first color sub-pixel, and the second matrix is ​​the white balance coefficient matrix of the second color sub-pixel.

[0137] In an exemplary embodiment of the present disclosure, the image segmentation module includes: a segmentation control module, which is used to segment the image to be processed based on the color temperature sensor array of the terminal to obtain the multiple image blocks; wherein the number of the image blocks is the same as the number of color temperature sensors in the color temperature sensor array, and each image block corresponds to a color temperature sensor.

[0138] In an exemplary embodiment of the present disclosure, the coefficient acquisition module includes: a first global coefficient acquisition module, which is used to obtain global attribute parameters according to the attribute parameters of each image block obtained by the color temperature sensor array, and process the global attribute parameters to obtain the global white balance coefficient.

[0139] In an exemplary embodiment of the present disclosure, the global white balance coefficient acquisition module includes: a second global coefficient acquisition module, configured to average the local white balance coefficients of all image blocks to acquire the global white balance coefficient.

[0140] In an exemplary embodiment of the present disclosure, the attribute parameter is a spectrum; the first global coefficient acquisition module includes: a spectral coefficient acquisition module, which is used to average the spectra of each image block acquired according to the color temperature sensor array to obtain global spectral data, and predict the global spectral data to obtain the global white balance coefficient.

[0141] In an exemplary embodiment of the present disclosure, the attribute parameter is color temperature; the first global coefficient acquisition module includes: a color temperature coefficient acquisition module, which is used to average the color temperatures of each image block acquired according to the color temperature sensor array to obtain a global color temperature value, and linearly interpolate the global color temperature value in the color temperature value sequence to obtain the global white balance coefficient.

[0142] It should be noted that the specific details of each part of the above-mentioned image white balance processing device have been described in detail in the implementation method of the image white balance processing method. The undisclosed details can be found in the implementation method of the method part, and will not be repeated here.

[0143] In the embodiment of the present disclosure, an image module is also provided. Fig.13 As shown in , the image module 1300 may include:

[0144] Camera module 1301; and

[0145] A color temperature sensor array 1302 is provided on one side of the camera module and is used to determine a global white balance coefficient of an image to be processed and a local white balance coefficient of an image block in the image to be processed;

[0146] An interpolation component 1303, configured to obtain a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient;

[0147] The image generation component 1304 is used to perform white balance processing on the image to be processed according to the target coefficient matrix to generate a target image corresponding to the image to be processed.

[0148] In the disclosed embodiment, the camera module may be a rear camera module, and the camera module may include at least one camera, for example, a main camera, a telephoto camera, a wide-angle camera, a macro camera, or any one or a combination thereof. The specific location and arrangement order of the multiple cameras may be determined according to actual needs and are not specifically limited here.

[0149] The color temperature sensor array can be set on one side of the camera module, specifically on one side of any camera. The position of the color temperature sensor array can be determined based on the calibration results in the actual application scenario. Fig.14 As shown in , the color temperature sensor array 1404 can be arranged below the camera 1403 in the terminal 1400 or beside the camera 1403, or beside the camera 1401 or the camera 1402 (for example, the left side or the right side). The camera module and the color temperature sensor array can be arranged adjacent to each other or separated by some distance. The color temperature sensor can use a silicon-based detector or a short-wave infrared detector to generate a white balance coefficient matrix based on the attribute parameters obtained by the color temperature sensor.

[0150] Through the color temperature sensor array in the imaging module, the image to be processed can be divided into multiple image blocks, and then the global white balance coefficient and the local white balance coefficient are generated based on the attribute parameters of the image block corresponding to each color temperature sensor. The specific method of generating the global white balance coefficient and the local white balance coefficient is the same as that in the above step S220, and will not be repeated here.

[0151] Furthermore, the interpolation component can be used to restrict the value range of the local white balance coefficient based on the global white balance coefficient, and perform partition interpolation and other operations on the local white balance coefficient to obtain a target coefficient matrix; and the parameters of the first color sub-pixel and the second color sub-pixel of each pixel point in the image to be processed are corrected according to the first matrix and the second matrix of the target coefficient matrix to generate a target image corresponding to the image to be processed. The specific method is the same as that in step S230 and step S240, and will not be repeated here.

[0152] In the disclosed embodiment, since one color temperature sensor in the color temperature sensor array included in the imaging module can correspond to one image block, the color temperature sensor array can obtain the attribute parameters of each image block, so the local white balance coefficient and the global white balance coefficient of each image block can be determined more accurately through the attribute parameters of each image block obtained by the color temperature sensor array, and the target coefficient matrix for white balance processing can be determined based on the local white balance coefficient and the global white balance coefficient, thereby improving the accuracy of the white balance coefficient. Furthermore, the parameters of the first color sub-pixel and the second color sub-pixel of each pixel point in the image to be processed can be adjusted and corrected through the obtained target coefficient matrix, thereby obtaining a target image after white balance processing, thereby improving the image white balance effect and image quality.

[0153] The exemplary embodiment of the present disclosure further provides an electronic device. The electronic device may be the above-mentioned terminal 101. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned image white balance processing method by executing the executable instructions.

[0154] Below Fig.15 The mobile terminal 1500 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig.15 The construction in can also be applied to fixed type equipment.

[0155] like Fig.15 As shown, the mobile terminal 1500 may specifically include: a processor 1501, a memory 1502, a bus 1503, a mobile communication module 1504, an antenna 1, a wireless communication module 1505, an antenna 2, a display screen 1506, a camera module 1507, an audio module 1508, a power module 1509 and a sensor module 1510.

[0156] The processor 1501 may include one or more processing units, for example: the processor 1501 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The image denoising processing method in this exemplary embodiment may be executed by an AP, a GPU or a DSP. When the method involves processing related to a neural network, it may be executed by an NPU, for example, the NPU may load neural network parameters and execute algorithm instructions related to the neural network. Exemplarily, the image to be processed may be divided into blocks to obtain a plurality of image blocks; the global white balance coefficient of the image to be processed is determined, and the local white balance coefficient of each image block is determined; according to the global white balance coefficient and the local white balance coefficient, a target coefficient matrix corresponding to each pixel of the image to be processed is obtained by interpolation; the parameters of each pixel in the image to be processed are corrected according to the target coefficient matrix to generate a target image corresponding to the image to be processed.

[0157] The encoder can encode (i.e. compress) an image or video to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data. The mobile terminal 1500 can support one or more encoders and decoders, for example: image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and video formats such as MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, and HEVC (High Efficiency Video Coding).

[0158] The processor 1501 may be connected to the memory 1502 or other components via a bus 1503 .

[0159] The memory 1502 may be used to store computer executable program codes, which may include instructions. The processor 1501 executes various functional applications and data processing of the mobile terminal 1500 by running the instructions stored in the memory 1502. The memory 1502 may also store application data, such as images, videos, and other files.

[0160] The communication function of the mobile terminal 1500 can be implemented by the mobile communication module 1504, antenna 1, wireless communication module 1505, antenna 2, a modulation and demodulation processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1504 can provide 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1500. The wireless communication module 1505 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 1500.

[0161] The display screen 1506 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 1507 is used to implement shooting functions, such as shooting images, videos, etc., and the camera module may include a color temperature sensor array. The audio module 1508 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 1509 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 1510 may include one or more sensors for implementing corresponding sensing detection functions. For example, the sensor module 1510 may include an inertial sensor, which is used to detect the motion posture of the mobile terminal 1500 and output inertial sensing data.

[0162] It should be noted that a computer-readable storage medium is also provided in an embodiment of the present disclosure. The computer-readable storage medium may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0163] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0164] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0165] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0166] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0167] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0168] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0169] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims. It should be understood that the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for image white balance processing, characterized in that: include: Acquire an image to be processed, and divide the image to be processed into blocks to obtain a plurality of image blocks; Determining a global white balance coefficient of the image to be processed, and determining a local white balance coefficient of each of the image blocks; Acquire a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient; Performing white balance processing on the image to be processed according to the target coefficient matrix to generate a target image corresponding to the image to be processed; Wherein, obtaining a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient includes: Limiting the value range of the local white balance coefficient by using the global white balance coefficient to obtain a limited local white balance coefficient; The restricted local white balance coefficients of the first area, the second area and the third area of ​​the multiple image blocks are respectively subjected to partition interpolation using different interpolation methods to determine a target coefficient matrix.

2. The image white balance processing method according to claim 1, characterized in that: The step of limiting the value range of the local white balance coefficient by using the global white balance coefficient to obtain the limited local white balance coefficient includes: The local white balance coefficient is adjusted within a preset floating range based on the global white balance coefficient to determine the limited local white balance coefficient.

3. The image white balance processing method according to claim 1, characterized in that: The method further comprises: Determine a first region, a second region, and a third region; wherein the image block includes a plurality of sub-image blocks; the sub-image blocks located at the corners of the image to be processed form the first region, the sub-image blocks located at the edges of the image to be processed form the second region, and the sub-image blocks located in the middle of the image to be processed form the third region.

4. The image white balance processing method according to claim 1, characterized in that: The interpolating the first region, the second region, and the third region respectively by using different interpolation methods to determine the target coefficient matrix includes: Determining an interpolation coefficient corresponding to the sub-image block of the first area based on a restricted local white balance coefficient corresponding to the image block to which the sub-image block of the first area belongs; Performing linear interpolation on restricted local white balance coefficients corresponding to image blocks to which adjacent sub-image blocks in the second region belong, to determine interpolation coefficients corresponding to sub-image blocks in the second region; Bilinear interpolation is performed on restricted local white balance coefficients corresponding to image blocks to which adjacent sub-image blocks in the third region belong, and interpolation coefficients corresponding to sub-image blocks in the third region are determined to determine the target coefficient matrix.

5. The image white balance processing method according to claim 1, characterized in that: Before interpolating the restricted local white balance coefficients to determine the target coefficient matrix, the method further includes: Performing median filtering on the restricted local white balance coefficient to obtain a median filtering result; The median filtering result is subjected to mean filtering for smoothing.

6. The image white balance processing method according to claim 1, characterized in that: The step of performing white balance processing on the image to be processed according to the target coefficient matrix to generate a target image corresponding to the image to be processed includes: The parameters of the first color sub-pixel and the second color sub-pixel of each pixel point in the image to be processed are corrected according to the target coefficient matrix to obtain the target image.

7. The image white balance processing method according to claim 6, characterized in that: The target coefficient matrix includes a first matrix and a second matrix; and the step of correcting the parameters of the first color sub-pixel and the second color sub-pixel of each pixel point in the image to be processed according to the target coefficient matrix to obtain the target image includes: Correcting parameters of the first color sub-pixel according to a first matrix in the target coefficient matrix; Correcting parameters of the second color sub-pixel according to a second matrix in the target coefficient matrix; The first matrix is ​​a white balance coefficient matrix of a first color sub-pixel, and the second matrix is ​​a white balance coefficient matrix of a second color sub-pixel.

8. The image white balance processing method according to claim 1, characterized in that: The step of dividing the image to be processed into multiple image blocks comprises: The image to be processed is divided into blocks based on the color temperature sensor array of the terminal to obtain the multiple image blocks; wherein the number of the image blocks is the same as the number of color temperature sensors included in the color temperature sensor array, and each image block corresponds to a color temperature sensor.

9. The image white balance processing method according to claim 8, characterized in that: The step of determining a global white balance coefficient of the image to be processed includes: A global attribute parameter is obtained according to the attribute parameter of each image block obtained by the color temperature sensor array, and the global attribute parameter is processed to obtain the global white balance coefficient.

10. The image white balance processing method according to claim 1, characterized in that: The step of determining a global white balance coefficient of the image to be processed includes: The local white balance coefficients of all image blocks are averaged to obtain the global white balance coefficient.

11. The image white balance processing method according to claim 9, characterized in that: The attribute parameter is a spectrum; the step of obtaining a global attribute parameter according to the attribute parameter of each image block obtained by the color temperature sensor array, and processing the global attribute parameter to obtain the global white balance coefficient includes: The spectrum of each image block obtained according to the color temperature sensor array is averaged to obtain global spectrum data, and the global spectrum data is predicted to obtain the global white balance coefficient.

12. The image white balance processing method according to claim 9, characterized in that: The attribute parameter is color temperature; the step of obtaining a global attribute parameter according to the attribute parameters of each image block obtained by the color temperature sensor array, and processing the global attribute parameter to obtain the global white balance coefficient includes: The color temperatures of the image blocks acquired according to the color temperature sensor array are averaged to obtain a global color temperature value, and the global color temperature value is linearly interpolated in a color temperature value sequence to obtain the global white balance coefficient.

13. An image white balance processing device, characterized in that: include: An image segmentation module is used to obtain an image to be processed and segment the image to be processed into multiple image blocks; A coefficient acquisition module, used to determine the global white balance coefficient of the image to be processed, and determine the local white balance coefficient of each image block; An interpolation module, used for obtaining a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient; An image generation module, used for performing white balance processing on the image to be processed according to the target coefficient matrix to generate a target image corresponding to the image to be processed; Wherein, obtaining a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient includes: Limiting the value range of the local white balance coefficient by using the global white balance coefficient to obtain a limited local white balance coefficient; The restricted local white balance coefficients of the first area, the second area and the third area of ​​the multiple image blocks are respectively subjected to partition interpolation using different interpolation methods to determine a target coefficient matrix.

14. An image module, characterized in that: include: A camera module is used to shoot an object to be shot to obtain an image to be processed; A color temperature sensor array, provided on one side of the camera module, is used to divide the image to be processed into multiple image blocks, determine the global white balance coefficient of the image to be processed and the local white balance coefficient of each image block in the image to be processed; An interpolation component, configured to obtain a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient; An image generation component, used for performing white balance processing on the image to be processed according to the target coefficient matrix to generate a target image corresponding to the image to be processed; Wherein, obtaining a target coefficient matrix according to the global white balance coefficient and the local white balance coefficient includes: Limiting the value range of the local white balance coefficient by using the global white balance coefficient to obtain a limited local white balance coefficient; The restricted local white balance coefficients of the first area, the second area and the third area of ​​the multiple image blocks are respectively subjected to partition interpolation using different interpolation methods to determine a target coefficient matrix.

15. An electronic device, characterized in that: include: The imaging module as claimed in claim 14; processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the image white balance processing method described in any one of claims 1-12 by executing the executable instructions.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image white balance processing method according to any one of claims 1 to 12 is implemented.

Images