Image processing method and device, electronic equipment, and storage medium

Abstract

The embodiment of the present disclosure relates to an image processing method and device, electronic equipment and storage medium, and relates to the technical field of images. The image processing method comprises the following steps: acquiring a to-be-processed image, and dividing the to-be-processed image to obtain a plurality of image blocks; acquiring color temperature information of each image block, and determining local color temperature information of each image block through a color sensor array; determining target color temperature information of each image block according to the color temperature information and the local color temperature information, and acquiring a color correction gain matrix of the target color temperature information; processing color information of the image block according to the color correction gain matrix, acquiring target color information of the image block, and generating a target image corresponding to the to-be-processed image. The technical solution in the embodiment of the present disclosure can improve the accuracy of color correction.

Description

Technical Field

[0001] This disclosure relates to the field of imaging technology, and more specifically, to an image processing method and apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the process of image processing, it may be necessary to perform color correction on the image to improve image quality.

[0003] In related technologies, color correction utilizes color charts calibrated under different color temperature light sources, then the device determines the color temperature of the current scene in real time, and finally interpolates to calculate the required color correction matrix. This method has certain limitations, and the actual usage scenarios of the devices are complex and variable environments, which can lead to some deviation in accurate color temperature determination, resulting in an inaccurate color correction matrix and affecting image quality.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an image processing method and apparatus, electronic device, and computer-readable storage medium, thereby overcoming, at least to some extent, the problem of poor image correction accuracy caused by the limitations and defects of related technologies.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to a first aspect of this disclosure, an image processing method is provided, comprising: acquiring an image to be processed, and dividing the image to be processed into multiple image blocks; acquiring color temperature information of each image block, and determining local color temperature information of each image block through a color sensor array; determining target color temperature information of each image block based on the color temperature information and the local color temperature information, and acquiring a color correction gain matrix of the target color temperature information; processing the color information of the image blocks based on the color correction gain matrix to obtain target color information of the image blocks, thereby generating a target image corresponding to the image to be processed.

[0008] According to a second aspect of this disclosure, an image processing apparatus is provided, comprising: an image segmentation module for acquiring an image to be processed and segmenting the image to be processed into multiple image blocks; a color temperature acquisition module for acquiring color temperature information of each image block and determining local color temperature information of each image block through a color sensor array; a matrix determination module for determining target color temperature information of each image block based on the color temperature information and the local color temperature information, and acquiring a color correction gain matrix of the target color temperature information; and a color correction module for processing the color information of the image blocks according to the color correction gain matrix to acquire target color information of the image blocks, thereby generating a target image corresponding to the image to be processed.

[0009] According to a third aspect of this disclosure, an electronic device is provided, comprising: an imaging module including a color sensor array; a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image processing method of the first aspect and possible implementations thereof by executing the executable instructions.

[0010] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the image processing method of the first aspect described above and its possible implementations.

[0011] The technical solution provided in this disclosure, on the one hand, obtains the target color temperature information of each image block by acquiring local color temperature information through a color sensor array, and then determines the color correction gain matrix corresponding to the target color temperature information. This avoids deviations caused by the influence of the external environment on color temperature, improves the accuracy and stability of the target color temperature information, and thus improves the accuracy of the color correction gain matrix corresponding to the target color temperature information. This enhances the color correction capability and reliability of the color correction module, and improves image quality and image effect. On the other hand, calculating the color correction gain matrix of each image block using the target color temperature information of each image block allows for independent control of each image block, enabling local or complete color correction through each image block, thus improving flexibility. Furthermore, it avoids the limitation of only being able to determine the color temperature of the current scene based on the device in real time, expanding the application scope and improving application convenience.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0014] Figure 1 A schematic diagram illustrates an application scenario where the image processing method of the present disclosure embodiments can be applied.

[0015] Figure 2 The diagram illustrates an image processing method according to an embodiment of the present disclosure.

[0016] Figure 3 A schematic diagram of an image block is shown in an embodiment of this disclosure.

[0017] Figure 4 The schematic diagram illustrates the process of determining target color temperature information in different ways in the embodiments of this disclosure.

[0018] Figure 5 This diagram illustrates the weighting variation of target color temperature information in an embodiment of the present disclosure.

[0019] Figure 6 This diagram illustrates a color correction gain matrix for determining target color temperature information in an embodiment of the present disclosure.

[0020] Figure 7 A schematic diagram illustrating color mapping according to an embodiment of the present disclosure is shown.

[0021] Figure 8 The schematic diagram illustrates the structure of the image signal processor in an embodiment of this disclosure.

[0022] Figure 9 The schematic diagram illustrates the image color processing flow in an embodiment of this disclosure.

[0023] Figure 10 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0024] Figure 11 A block diagram of an electronic device according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0026] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] Current technical solutions perform color correction in the full RGB domain of the ISP and then use 2D / 3D LUTs (2D / 3D Lookup Tables) for color adjustment. Color correction utilizes a color chart calibrated under different color temperature light sources, then the device determines the current scene's color temperature in real time, and finally interpolates to calculate the required color correction matrix. The 2D / 3D LUT maps color information within the corresponding color coordinates using a pre-defined mapping relationship to obtain the desired color effect.

[0028] For the color correction module, initial calibration can only be performed under standard laboratory lighting. Actual use scenarios for the equipment involve complex and variable environments, often resulting in some deviation in accurate color temperature determination. This leads to poor accuracy in the final color correction coefficient interpolation calculation. For 2D / 3D LUTs, pre-judgment of the actual scene colors is required when performing preset mapping processing in the color space. For example, specific colors need to be processed, such as skin tones on faces, green plants, red flowers, and blue skies. Therefore, this approach has limitations and makes it difficult to complete the aforementioned color mapping.

[0029] In order to solve the technical problems in the related technologies, this disclosure provides an image processing method that can be applied to application scenarios where images are processed during the taking of photos. Figure 1A schematic diagram of a system architecture for an image processing method and apparatus applicable to embodiments of the present disclosure is shown.

[0030] like Figure 1 As shown, terminal 101 can be a smart device with image processing capabilities, such as a smartphone, computer, tablet, smart speaker, smartwatch, in-vehicle device, wearable device, monitoring device, etc. The terminal may include a camera; the type of camera can be any type, as long as it can perform image processing. The number of cameras can be at least one, for example, one, four, etc., as long as they can take pictures. The image to be processed can be a captured image or each frame of a captured video.

[0031] In this embodiment, terminal 101 may include memory 102 and processor 103. The memory stores images, and the processor processes the images, such as performing white balance processing. Memory 102 may store an image 104 to be processed. Terminal 101 retrieves the image 104 to be processed from memory 102 and sends it to processor 103. Processor 103 divides the image to be processed into multiple image blocks, determines the local color temperature information and color temperature information of the image to be processed. Based on the color temperature information and the local color temperature information of each image block obtained by the color sensor array, target color temperature information for each image block is determined. Furthermore, a color correction gain matrix for the target color temperature information is determined, and the image to be processed is color-corrected using the color correction gain matrix to generate a target image 105.

[0032] It should be noted that the image processing method provided in this embodiment can be executed by terminal 101. Alternatively, the image processing method can be configured within the terminal.

[0033] Next, refer to Figure 2 The image processing methods in the embodiments of this disclosure will be described in detail.

[0034] In step S210, the image to be processed is acquired, and the image to be processed is divided into multiple image blocks.

[0035] In this embodiment, the image to be processed can be an image captured by the camera module of the terminal, or it can be each frame of a captured video. The image to be processed can also be an image or each frame of a video directly obtained from a photo album or other storage location. The terminal can be any of the following: a smartphone, digital camera, smartwatch, wearable device, in-vehicle device, or surveillance camera, as long as it can capture images of the object and perform image processing. A smartphone is used as an example here. The camera module can include at least one camera, such as a main camera, telephoto camera, wide-angle camera, macro camera, or a combination thereof. The image to be processed can be various types of images, such as moving images or still images, etc.

[0036] The image to be processed can be an RGB image, i.e., an RGB three-channel image. Each pixel in an RGB image is composed of three colors: RGB. When the terminal is in camera mode, the image captured by the camera module can be a RAW image. A RAW image is the raw image data information acquired by the camera module. In this embodiment, the captured image obtained by the terminal can be converted to obtain the image to be processed. For example, a general conversion algorithm can be used to convert the RAW format captured image to RGB format to obtain the image to be processed, thereby improving the convenience of subsequent processing.

[0037] After acquiring the image to be processed, it can be divided into multiple image blocks. Each image block can be a portion of the image to be processed, and the blocks do not overlap. Each image block can be the same size, and the number of image blocks can be determined based on the number of grid cells. For example, a grid region can be provided and applied to the image to be processed to divide it into multiple image blocks according to the grid region, with each image block corresponding one-to-one with a grid region. (See reference...) Figure 3 As shown, a grid can contain multiple image blocks 301. For example, grid 00 corresponds to image block 00, grid 01 corresponds to image block 01, and so on. The size of the grid area can be set according to actual needs and hardware structure, that is, set according to actual needs within the limits of the hardware structure. For example, it can be row-based and column-based. Based on this, the color sensor can be associated with image blocks and grids, and the image to be processed can be divided into row × column image blocks, with each image block corresponding to each grid. Under the same field of view, the multi-window spatial range represented by the grid area is consistent with the image to be processed, so that the grid area can cover the image to be processed. Each grid can correspond to one window, hence it can be called a multi-window system.

[0038] In this embodiment, a color sensor array may be included, comprising multiple color sensors 302 arranged in an array. The number of color sensors can be determined based on the number of image patches. Furthermore, the color sensor array can be combined with multi-window information, thus each color sensor can be a multi-window color sensor. Each grid represents one color sensor, and each color sensor corresponds to each image patch. Since the image to be processed is divided into multiple grids, a row × col color sensor array is formed spatially, creating a multi-window color sensor array.

[0039] The multi-window color sensor array is an independent sensor that can be positioned to one side of any camera in the camera module, close to the camera module. The camera module can be a rear-facing camera module, and it can include at least one camera, such as a main camera, telephoto camera, wide-angle camera, macro camera, or a combination thereof. The specific placement and arrangement order of the multiple cameras can be determined according to actual needs and are not specifically limited here. For example, the color sensor array can be positioned to the left of the telephoto camera, to the right of the main camera, or below the last camera in the at least one camera setup, etc. The camera module and the color sensor array can be placed adjacent to each other or separated by a certain distance. The specific position of the color sensor array can be determined based on the calibration results during the actual application process or according to actual needs; it is not specifically limited here.

[0040] It should be noted that each module in the image signal processor can divide the image to be processed into blocks, and the resulting image blocks can be consistent with those obtained by dividing the image according to the grid, to ensure consistency and accuracy. For example, the color correction module can divide the image into multiple window regions to obtain the first image block; the 2D / 3D lookup table module can also divide the image into multiple window regions to obtain the second image block. The image division method is the same between different modules, and the first and second image blocks are basically consistent with the image blocks divided in step S210, to ensure consistency and accuracy between image blocks.

[0041] The multi-window area information of the color sensor array is designed to correspond with the color correction module. The window size of the color correction module should be equal to, slightly larger than, or smaller than the multi-window area of ​​the color sensor array, that is, the number of rows and columns of the two are the same.

[0042] Next, continue to refer to Figure 2 As shown, in step S220, the color temperature information of each image block is obtained, and the local color temperature information of each image block is determined by the color sensor array.

[0043] In this embodiment of the disclosure, when the color of light emitted by a light source is consistent with the color of light radiated by a black body (such as platinum) at a certain temperature, the temperature of the black body at this time is expressed as the color temperature of the light source, that is, the color temperature cct (correlated color temperature).

[0044] First, the color temperature information of each image patch can be obtained through an Image Signal Processor (ISP), specifically represented by cct1. In addition, the color temperature of local areas of the current scene can be calculated using a color sensor array, i.e., obtaining the local color temperature information of each image patch, represented by cct2. The color sensor can be a sensor used to detect scene color, color temperature, spectrum, and other related information. It can be used to detect the color information of the objects corresponding to each image patch and the color temperature information of the current scene. Objects can be any type of object contained in each image patch, such as objects, people, etc. The spectrum refers to different wavelengths, and the color temperature is the response curve for different wavelengths. Wavelength corresponds to color, so the spectrum, color temperature, and color are interrelated. The color temperature information and the local color temperature information can be the same or different, depending on the actual acquisition results.

[0045] In step S230, the target color temperature information of each image block is determined based on the color temperature information and the local color temperature information, and the color correction gain matrix of the target color temperature information is obtained.

[0046] In this embodiment of the disclosure, after obtaining color temperature information and local color temperature information, the target color temperature information of an image block can be determined based on the combined color temperature information and local color temperature information. For example, the color temperature information can be compared with the local color temperature information to determine the difference information; based on the comparison result between the difference information and a threshold parameter, different methods are selected to obtain the target color temperature information. Specifically, the difference information between the color temperature information and the local color temperature information can be calculated based on the absolute value of the difference between them. Further, the difference information can be compared with a threshold parameter to obtain a comparison result, and based on the comparison result between the difference information and the threshold parameter, different methods are selected to determine the target color temperature information of each image block. The threshold parameter may include a first threshold and a second threshold, where the first threshold Th1 is less than the second threshold Th2.

[0047] Figure 4 The diagram illustrates a flowchart of different methods for determining target color temperature information. (See reference) Figure 4 As shown, the main steps include:

[0048] In step S410, it is determined whether the difference information is less than the first threshold; if yes, proceed to step S420; if no, proceed to step S430.

[0049] In step S420, if the difference information is less than the first threshold, the target color temperature information is determined based on the color temperature information;

[0050] In step S430, it is determined whether the difference information is less than the second threshold; if yes, proceed to step S440; if no, proceed to step S450.

[0051] In step S440, if the difference information is greater than a first threshold and less than a second threshold, the color temperature information is adjusted using local color temperature information to determine the target color temperature information;

[0052] In step S450, if the difference information is greater than the second threshold, the target color temperature information is determined based on the local color temperature information.

[0053] In this embodiment of the disclosure, as the difference information changes, the weight of the target color temperature information in the total color temperature information gradually decreases, as detailed in the following reference. Figure 5 The weighting change diagram is shown below. When the difference information is greater than the first threshold, the target color temperature information has a weight of 1 in the total color temperature information. When the difference information is between the first and second thresholds, the weight of the target color temperature information gradually decreases. When the difference information is greater than the second threshold, the weight of the target color temperature information is 0, that is, the target color temperature information has a weight of 1 in the local color temperature information acquired by the color sensor array.

[0054] Based on this, if the difference between the two is less than a first threshold, the target color temperature information for each image block can be the color temperature information obtained by the image signal processor. In this case, the color sensor array does not need to correct the color temperature information obtained by the image signal processor. Therefore, the color temperature information is directly used as the target color temperature information for determining each image block.

[0055] If the difference between the two is greater than the first threshold and less than the second threshold, the target color temperature information of each image block can be determined jointly based on the color temperature information obtained by the color sensor array and the image signal processor. That is, the local color temperature information obtained by the color sensor array needs to be adjusted for the color temperature information obtained by the image signal processor, i.e., the color temperature information is interpolated by the local color temperature information obtained by the color sensor array. For example, the target color temperature information of each image block can be obtained by interpolating all color temperature information calculated in different ways. Interpolation can be a weighted summation operation. Among them, the weight parameters corresponding to the color temperature information obtained in different ways are also different, but the sum of all weight parameters is 1. Based on this, all color temperature information can be fused according to the weight parameters. For example, the target color temperature information can be obtained by weighted fusion of the local color temperature information and the color temperature information obtained by the color sensor array according to the corresponding weight parameters, as shown in formula (1):

[0056] cct=cct1*w1+cct2*w2 formula (1)

[0057] Here, cct1 represents the color temperature information acquired by the image signal processor, and cct2 represents the local color temperature information of each image block acquired by the color sensor array.

[0058] Adjusting the color temperature information by using local color temperature information obtained from a color sensor array can improve the accuracy of the target color temperature information for each image block, and also avoids the influence of the environment on the color temperature in related technologies, thus improving stability.

[0059] If the difference between the two is greater than the second threshold, the target color temperature information of each image block can be determined based on the local color temperature information obtained by the color sensor array.

[0060] In this embodiment of the disclosure, by comparing the difference between the local color temperature information and the color temperature information obtained by the color sensor array with the first threshold and the second threshold, and using different methods to obtain the target color temperature information of each image block based on the comparison result, the accuracy of the target color temperature information corresponding to each image block can be improved.

[0061] After determining the target color temperature information for each image block, this information can be smoothed to achieve a smooth transition between the target color temperature information of all image blocks, reducing the abrupt changes caused by the different target color temperature information applied to different local blocks represented by each image block. This smoothing process can be achieved through low-pass filtering. Low-pass filtering refers to a filtering method that allows low-frequency signals to pass normally, while blocking or attenuating high-frequency signals exceeding a set threshold. The degree of blocking or attenuation varies depending on the frequency, the filtering procedure, or the filtering objective.

[0062] Next, based on the target color temperature information of each image block, a color correction gain matrix representing the target color temperature information of each image block can be obtained to achieve color correction. Color correction refers to correcting the difference between the current color and the target color of an image using a 24-color chart.

[0063] Color correction refers to changing the color values ​​of all pixels in an image using the same method to achieve different display effects. When image acquisition systems acquire digital images, due to factors such as ambient lighting or human intervention, the acquired images often differ significantly from the original images. Color correction can reduce this difference to some extent. Color correction can be implemented using a color correction gain matrix. The color correction gain matrix is ​​used to correct the color parameters of each pixel to adjust it to the target color parameters. It is mainly used to convert white balance processed image data to the standard RGB color space. However, the RGB data after color correction is still linear and requires Gamma processing to convert the image data to the sRGB space, which is closer to the human eye's perception.

[0064] Specifically, the color correction gain matrix can be determined based on the target color temperature information of each image patch and the calibration light source result of the image signal processor. The calibration light source result refers to the color temperature information acquired by the image signal processor. In some embodiments, each image patch has a corresponding gain matrix for its color temperature information, and the gain matrices of each image patch can be the same or different. Furthermore, each image patch also has a corresponding gain matrix for its local color temperature information. The gain matrices for both the calibrated color temperature information and the local color temperature information can be, for example, 3×3 matrices.

[0065] Based on this, the gain matrix of the color temperature information of each image patch and the gain matrix of the local color temperature information can be interpolated to obtain the color correction gain matrix of the target color temperature information corresponding to each image patch. For example, the gain matrix of the color temperature information and the gain matrix of the local color temperature information can be fused according to corresponding weight parameters. This fusion can be achieved through a weighted summation operation. The weight parameters are negatively correlated with the distance from the corresponding color temperature information to the target color temperature information; that is, the greater the distance from the corresponding color temperature information to the target color temperature information, the smaller the weight parameter. The weight parameters of the gain matrix of each color temperature information can be determined based on the distance from another color temperature information to the target color temperature information. For example, refer to... Figure 6 As shown in the figure, the gain matrix of the color temperature information of point A (2000K) is A1 and its weight parameter is m2 / m1+m2. The gain matrix of the local color temperature information of point B (3000K) is B1 and its weight parameter is m1 / m1+m2. Then the color correction gain matrix of the target color temperature information at point C can be expressed as m2 / m1+m2*A1+m1 / m1+m2*B1.

[0066] After obtaining the color correction gain matrix for each image patch, the color correction gain matrix for each image patch can be smoothed again. Smoothing can be done through low-pass filtering, thus obtaining the color correction gain matrix for each image patch. The color correction gain matrix can be a red-green-blue gain matrix.

[0067] Furthermore, the color correction gain matrix can be applied to the image to be processed. For example, a target image block can be determined from multiple image blocks, and the color correction gain matrix of each image block contained within the target image block can be multiplied with the color parameters of the pixels of each image block to adjust the color parameters of the pixels of each image block to the required target color parameters, making the image color more consistent with actual needs and improving image quality. The target image block can be part or all of the multiple image blocks, depending on the specific requirements.

[0068] In this embodiment, by calculating the color correction gain matrix for each image block, each image block can be independently controlled, achieving local or complete color correction through each block, thus improving flexibility. Furthermore, by obtaining the target color temperature information for each image block using local color temperature information acquired from the color sensor array, and then determining the corresponding color correction gain matrix, deviations caused by the influence of external environmental color temperature on color information are avoided. This improves the accuracy and stability of the target color temperature information, thereby enhancing the accuracy of the color correction gain matrix and improving the color correction capability and reliability of the color correction module.

[0069] In some scenarios, the raw images captured by the camera lack overall vibrancy. Furthermore, it's desirable to capture images of different scenes, such as greenery, grass, flowers, blue skies, white clouds, beaches, buildings, and animals. Specific transformations of the hue and saturation of certain colors can be made. For example, when capturing greenery, a deeper green might be desired while other colors remain unchanged. In such cases, color mapping can be performed using a 2D / 3D LUT. Color mapping refers to applying color mapping to the original image or video image based on a pre-adjusted color mapping table. The color mapping table is a three-dimensional lookup table; inputting the IN_RGB color components, the 3D LUT directly outputs the corresponding OUT_RGB color components. In this embodiment, the color sensor array can also be associated with a 2D / 3D LUT (2D / 3D lookup table) to enhance its capabilities. It should be noted that the window size of the 2D / 3D lookup table module should be equal to, larger than, or smaller than the multi-window area of ​​the color sensor. Furthermore, the method of dividing the image blocks and the number of resulting image blocks are essentially the same.

[0070] Based on this, the method further includes: combining the local color information corresponding to the local color temperature information obtained by the color temperature sensor array, and the reference color information to perform color mapping on the color information of the image block to obtain the target color information. Since color temperature is a response curve for different wavelengths, and wavelength corresponds to color, color temperature and color are interrelated. For the color sensor array, each color sensor can obtain the local color temperature information and local color information of the corresponding image block, and the local color information corresponds to the local color temperature information. Alternatively, it can be understood that the local color information is the color information presented under the local color temperature information, and the local color information may change with changes in the local color temperature information.

[0071] In some embodiments, color mapping can be performed by combining local color information acquired by the color sensor array and reference color information acquired by the lookup table module to adjust the color information to the target color information. Color mapping can be used to implement color correction to adjust color parameters, such as adjusting the values ​​of the three RGB sub-pixels. In this embodiment, color mapping can be performed by combining the parameter mapping relationship between the color sensor array and the lookup table module itself. Specifically, color mapping by combining local color information acquired by the color sensor array and reference color information acquired by the lookup table module can be performed by combining color threshold parameters and region type, and using different mapping methods to determine the target color information. The color threshold parameters can be a first color threshold and a second color threshold, wherein the first color threshold is different from the second color threshold, the second color threshold is different from the second color threshold, and the first color threshold is less than the second color threshold.

[0072] For example, the local color information can be compared with the reference color information to determine the color difference information; based on the comparison result between the color difference information and the color threshold parameter, different mapping methods can be selected to obtain the target color information. Specifically, the color difference information between the reference color information and the local color information can be calculated based on the absolute value of the difference between them. Further, the color difference information can be compared with the color threshold parameter to obtain a comparison result, and based on this comparison result, different mapping methods can be selected to determine the target color information for each image block. Specifically, this can involve determining the target color information for each pixel in the image.

[0073] In this embodiment of the disclosure, when determining that the region type of the image block is the target region type, the target color information can be determined by combining the comparison results between color difference information and color threshold parameters, and by combining the color threshold parameters and the region type, using different mapping methods composed of local color information and reference color information. Figure 7 The flowchart illustrating the color mapping process is shown in the image. Figure 7As shown, the main steps include:

[0074] In step S710, it is determined whether the region type of the image patch is a target type.

[0075] The target type refers to the type of region whose color needs to be adjusted. This can be determined based on the type of objects contained within the region, its color information, or whether user interaction has been received. For example, if an image patch contains objects such as human faces, skin tones, blue skies, or greenery, then the region type of the image patch can be determined to be the target type.

[0076] In step S720, it is determined whether the color difference information is less than the first color threshold; if yes, proceed to step S730; if no, proceed to step S740.

[0077] In step S730, if the color difference information is less than the first color threshold, the reference color information is mapped through a parameter mapping relationship to obtain the target color information.

[0078] In this step, the parameter mapping relationship can be a lookup table (2D / 3D-Look-Up-Table, abbreviated as 2D / 3DLUT), such as a color map table. For example, the color information of an image patch can be input into the color map table, and the output will be the target color information corresponding to the color map table. The intensity of color enhancement can be the intensity set by the lookup table itself, or it can be adjusted according to actual needs. For instance, the current RGB color components of a pixel in an image patch can be input into the color map table, and the corresponding RGB color mapping components will be output through the color map table. When performing color mapping through the map table, since an image patch can contain multiple pixels, each pixel can be adjusted according to the corresponding color map table to obtain the target color information after color adjustment.

[0079] In addition, the parameter mapping relationship can be a non-color mapping table. For example, the infrared information of the input image can be output as pseudo-color information through a non-color mapping table. This disclosure does not specifically limit the type of image parameter mapping relationship; the appropriate parameter mapping relationship can be selected according to the image parameter correction requirements.

[0080] In step S740, it is determined whether the color difference information is less than the second color threshold; if yes, proceed to step S750; if no, proceed to step S760.

[0081] In step S750, if the color difference information is greater than the first color threshold and less than the second color threshold, the target color information after color mapping is determined by fusing local color information and reference color information.

[0082] In this step, if the color difference information falls between two color threshold parameters, the target color information can be determined jointly based on the local color information and the reference color information. For example, the local color information and the reference color information can be weighted and fused according to their respective weights to obtain the target color information. As the difference information increases, the weight of the local color information gradually increases, while the weight of the reference color information gradually decreases.

[0083] In step S760, if the color difference information is greater than the second color threshold, the target color information is determined based on the local color information.

[0084] If the color difference information is greater than the second color threshold, it means that the mapped color information may be inaccurate. In this case, the local color information output by the color sensor array needs to be used as the target color information, that is, the local color information has a larger weight.

[0085] Furthermore, a smooth transition can be performed on the target color information obtained after color mapping for each image patch. This smooth transition can be achieved through low-pass filtering, making the transition between different image patches in the spatial domain smoother and avoiding abrupt color changes between patches, thus improving image quality. Spatial domain processing refers to pixel-level processing.

[0086] It's worth noting that during image processing, for real-time processing methods such as video or previews, temporal smoothing can be performed. At each time step t, an image can be input into the image signal processing system for processing; each time step or frame corresponds to one image. During this process, temporal smoothing filtering can be applied to each image. Temporal smoothing filtering of each image can be understood as smoothing filtering along the image's temporal sequence. Specifically, smoothing filtering can employ IIR filtering in the temporal domain, where the output is calculated as I = A*w + B*(1-w). Here, I represents the current frame, i.e., the output of Frame t; A is the data or parameters of the current frame; w is the weight of the current frame; B is the data or parameters of the previous frame (Frame-1) adjacent to the current frame on the time axis; and 1-w is the weight of Frame-1. This method allows for temporal (image sequence) smoothing to reduce the differences between different times and avoid abrupt changes.

[0087] Figure 8 A schematic diagram of the image signal processor is shown for reference. Figure 8As shown, the image signal processor may include a color sensor array 801, a grid 802, an algorithm module 803, and may also include modules such as a color correction module 804, a tone mapping module 805, and a 2D / 3D LUT 806.

[0088] refer to Figure 8 As shown, the grid information acquired by the color sensor array is correlated with the grid acquired by the color correction module and the 2D / 3D LUT to ensure that the divided image patches are basically consistent. The local color temperature information output by the algorithm module is input to the color correction module, and the local color information output by the algorithm module is input to the 2D / 3D LUT. The color correction module and the 2D / 3D LUT are usually located in the second stage of the ISP processing flow, namely the RGB domain stage.

[0089] Based on the aforementioned hardware structure, the color sensor array can acquire the local color temperature information of each image block. By comparing the difference between the local color temperature information and the color temperature information of the image block acquired by the image signal processor with a threshold parameter, different methods are used to acquire the target color temperature information of each image block. Then, based on the gain matrix of the local color temperature information and the gain matrix of the color temperature information, the color correction gain matrix corresponding to the target color temperature information of each image block is obtained. Based on the color correction gain matrix corresponding to the target color temperature information of each image block, the color information of the image block is corrected to obtain the target color information of the image block, thereby generating the target image corresponding to the image to be processed.

[0090] In addition to acquiring the local color temperature and local color information of each image patch, the color sensor array can also determine whether the region type of the image patch is the target type. When the region type of the image patch is the target type, the target color information corresponding to the color information of the image patch can be determined by different mapping methods based on the color threshold parameter, the comparison result between the color difference information of the local color information and the reference color information of each image patch, and the region type of each image patch. If the region type of the image patch does not belong to the target type, color mapping is not required.

[0091] In this embodiment of the disclosure, by introducing a color sensor array to obtain local color temperature information and local color information, the color adjustment capability of the color correction module and the 2D / 3D LUT module can be improved, thereby improving the accuracy of the target color information.

[0092] Figure 9 The flowchart for acquiring the target image is illustrated in the diagram. Figure 9 As shown, the main steps include:

[0093] In step S901, the image to be processed 910 is acquired.

[0094] In step S902, the image to be processed is divided into multiple image blocks 920.

[0095] In step S903, the color temperature information 930 of each image block is obtained.

[0096] In step S904, local color temperature information 950 for each image block is acquired based on the color sensor array 940.

[0097] In step S905, target color temperature information 960 of the image block is generated based on color temperature information 930 and local color temperature information 950.

[0098] In step S906, the color correction gain matrix 970 of the target color temperature information 960 is obtained based on the gain matrix 971 of the color temperature information 930 and the gain matrix 972 of the local color temperature information 950.

[0099] In step S907, the color correction gain matrix is ​​used to correct the color of the image blocks in the image to be processed to obtain the target image 900.

[0100] In step S908, different mapping methods 990 are obtained by combining the local color information 980 of the color sensor array to map the color information of the image block and obtain the target image 900.

[0101] In step S909, the target image 900 is output.

[0102] In this embodiment, by introducing a color sensor array, local color temperature information of each image block can be obtained based on the multi-window information of the color sensor array. Furthermore, the color temperature information of each image block can be adjusted based on its local color temperature information to obtain the target color temperature information for each image block. Then, a color correction gain matrix for the target color temperature information is obtained based on the gain matrix of the color temperature information and the gain matrix of the local color temperature information. By calculating the color correction gain matrix for each image block, each image block can be independently controlled, achieving local or complete color correction, thus improving flexibility. In addition, obtaining the target color temperature information for each image block using the local color temperature information acquired by the color sensor array, and then determining the corresponding color correction gain matrix, avoids deviations caused by the influence of external environmental color temperature on color information, improving the accuracy and stability of the target color temperature information, thereby improving the accuracy of the color correction gain matrix and enhancing the color correction capability and reliability of the color correction module. Furthermore, by selecting an appropriate mapping method based on the local color information acquired by the color sensor array to map the color information to obtain the target color information, the accuracy and realism of the target color information are improved, and color richness is enhanced. By combining the target color temperature information of each image block with the local color temperature information acquired by the color sensor array, the color processing of the image to be processed can be performed more accurately from multiple dimensions such as global, local, and image content. This can improve the effect and realism of block white balance, achieve smooth transition between image blocks, improve the accuracy of image processing, and improve image quality.

[0103] This disclosure provides an image processing apparatus, with reference to... Figure 10 As shown, the image processing apparatus 1000 may include:

[0104] The image segmentation module 1001 is used to acquire the image to be processed and segment the image to be processed into multiple image blocks;

[0105] The color temperature acquisition module 1002 is used to acquire the color temperature information of each image block and determine the local color temperature information of each image block through the color sensor array;

[0106] The matrix determination module 1003 is used to determine the target color temperature information of each image block based on the color temperature information and the local color temperature information, and to obtain the color correction gain matrix of the target color temperature information.

[0107] The color correction module 1004 is used to process the color information of the image block according to the color correction gain matrix, obtain the target color information of the image block, and generate the target image corresponding to the image to be processed.

[0108] In one exemplary embodiment of this disclosure, the matrix determination module includes: a difference information determination module, configured to compare the color temperature information with the local color temperature information to determine difference information; and a method determination module, configured to select different methods to obtain the target color temperature information based on the comparison result between the difference information and a threshold parameter.

[0109] In an exemplary embodiment of this disclosure, the mode determination module includes: a first determination module, configured to determine target color temperature information based on color temperature information if the difference information is less than a first threshold; a second determination module, configured to adjust the color temperature information using local color temperature information to determine the target color temperature information if the difference information is greater than the first threshold and less than a second threshold; and a third determination module, configured to determine the target color temperature information based on the local color temperature information if the difference information is greater than the second threshold.

[0110] In one exemplary embodiment of this disclosure, the second determining module includes: an interpolation acquisition module, configured to interpolate the color temperature information based on the local color temperature information to obtain the target color temperature information.

[0111] In one exemplary embodiment of this disclosure, the matrix determination module includes: a matrix interpolation module, used to interpolate the gain matrix of the color temperature information of each image block and the gain matrix of the local color temperature information to obtain the color correction gain matrix of the target color temperature information corresponding to each image block.

[0112] In one exemplary embodiment of this disclosure, the matrix interpolation module includes: a fusion module, configured to fuse the gain matrix of the color temperature information with the gain matrix of the local color temperature information according to corresponding weight parameters to obtain the color correction gain matrix.

[0113] In one exemplary embodiment of this disclosure, the apparatus further includes: a color mapping module, configured to combine local color information corresponding to local color information obtained by the color temperature sensor array and reference color information to perform color mapping on the color information of the image block to obtain target color information.

[0114] In one exemplary embodiment of this disclosure, the color mapping module includes a mapping method determination module, configured to determine the target color information using different mapping methods based on a color threshold parameter, a comparison result between the color difference information of the local color information and the reference color information of each image block, and the region type of each image block.

[0115] In an exemplary embodiment of this disclosure, the mapping method determination module is configured to: if the color difference information is less than a first color threshold and the region type is a target type, map the reference color information through a parameter mapping relationship to obtain the target color information; if the color difference information is greater than the first color threshold and less than a second color threshold, and the region type is a target type, fuse the local color information and the reference color information to obtain the target color information; if the color difference information is greater than the second color threshold and the region type is a target type, determine the target color information based on the local color information.

[0116] It should be noted that the specific details of each part of the above-mentioned image processing apparatus have been described in detail in the implementation of the image processing method section. For any undisclosed details, please refer to the implementation of the method section, and therefore will not be repeated here.

[0117] An exemplary embodiment of this disclosure also provides an electronic device. This electronic device may be the terminal 101 described above. Generally, the electronic device may include a processor and a memory, the memory being used to store executable instructions of the processor, the processor being configured to perform the image processing method described above by executing the executable instructions.

[0118] The following is based on Figure 11 Taking the mobile terminal 1100 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 11 The structure can also be applied to fixed types of equipment.

[0119] like Figure 11 As shown, the mobile terminal 1100 may specifically include: a processor 1101, a memory 1102, a bus 1103, a mobile communication module 1104, an antenna 1, a wireless communication module 1105, an antenna 2, a display screen 1106, a camera module 1107, an audio module 1108, a power module 1109, and a sensor module 1110.

[0120] Processor 1101 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The image denoising method in this exemplary embodiment can be executed by an AP, GPU, or DSP. When the method involves neural network-related processing, it can be executed by an NPU. For example, the NPU can load neural network parameters and execute neural network-related algorithm instructions.

[0121] An encoder encodes (compresses) images or videos to reduce data size for easier storage or transmission. A decoder decodes (decompresses) the encoded data to restore the original image or video data. The mobile terminal 1100 can support one or more encoders and decoders, such as image formats like JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), and BMP (Bitmap), and video formats like MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, and HEVC (High Efficiency Video Coding).

[0122] The processor 1101 can be connected to the memory 1102 or other components via the bus 1103.

[0123] The memory 1102 can be used to store computer executable program code, which includes instructions. The processor 1101 executes various functional applications and data processing of the mobile terminal 1100 by running the instructions stored in the memory 1102. The memory 1102 can also store application data, such as images, videos, and other files.

[0124] The communication function of mobile terminal 1100 can be implemented through mobile communication module 1104, antenna 1, wireless communication module 1105, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1104 can provide 3G, 4G, 5G and other mobile communication solutions for mobile terminal 1100. Wireless communication module 1105 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1100.

[0125] The display screen 1106 is used to implement display functions, such as displaying user interfaces, images, and videos. The camera module 1107 is used to implement shooting functions, such as capturing images and videos, and may include a color sensor array. The audio module 1108 is used to implement audio functions, such as playing audio and capturing voice. The power module 1109 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring battery status. The sensor module 1110 may include one or more sensors to implement corresponding sensing and detection functions. For example, the sensor module 1110 may include an inertial sensor, which is used to detect the motion posture of the mobile terminal 1100 and output inertial sensing data.

[0126] It should be noted that the present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist alone and not assembled into the electronic device.

[0127] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0128] A computer-readable storage medium can be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0129] A computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments.

[0130] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0131] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0132] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0133] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims. It should be understood that this disclosure is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: The image to be processed is acquired, and the image to be processed is divided into multiple image blocks; The color temperature information of each image block is obtained, and the local color temperature information of each image block is determined by a color sensor array; The target color temperature information of each image block is determined based on the color temperature information and the local color temperature information, and the color correction gain matrix of the target color temperature information is obtained. The color information of the image block is processed according to the color correction gain matrix to obtain the target color information of the image block, so as to generate the target image corresponding to the image to be processed; The determination of the target color temperature information for each image block based on the color temperature information and the local color temperature information includes: The color temperature information is compared with the local color temperature information to determine the difference information; If the difference information is less than the first threshold, the color temperature information is determined as the target color temperature information for each image block; if the difference information is greater than the first threshold and less than the second threshold, the local color temperature information and the color temperature information are weighted and fused according to the weight parameters to obtain the target color temperature information; if the difference information is greater than the second threshold, the local color temperature information is determined as the target color temperature information. The process of obtaining the color correction gain matrix of the target color temperature information includes: Interpolate the gain matrix of the color temperature information of each image block and the gain matrix of the local color temperature information to obtain the color correction gain matrix of the target color temperature information corresponding to each image block.

2. The image processing method according to claim 1, characterized in that, The step of interpolating the gain matrix of the color temperature information of each image block and the gain matrix of the local color temperature information to obtain the color correction gain matrix of the target color temperature information corresponding to each image block includes: The color correction gain matrix is ​​obtained by fusing the gain matrix of the color temperature information with the gain matrix of the local color temperature information according to the corresponding weight parameters.

3. The image processing method according to claim 1, characterized in that, The method further includes: By combining the local color information corresponding to the local color temperature information obtained from the color temperature sensor array, and the reference color information, color mapping is performed on the color information of the image block to obtain the target color information.

4. The image processing method according to claim 3, characterized in that, The step of combining the local color information corresponding to the local color information obtained by the color temperature sensor array and the reference color information to perform color mapping on the color information of the image block to obtain target color information includes: Based on the color threshold parameter, the comparison results between the color difference information of the local color information and the reference color information of each image block, and the region type of each image block, the target color information is determined using different mapping methods.

5. The image processing method according to claim 4, characterized in that, The step of determining the target color information using different mapping methods based on a color threshold parameter, the comparison result between the color difference information of the local color information and the reference color information of each image block, and the region type of each image block includes: If the color difference information is less than the first color threshold and the region type is the target type, the reference color information is mapped through the parameter mapping relationship to obtain the target color information; If the color difference information is greater than the first color threshold and less than the second color threshold, and the region type is the target type, the local color information and the reference color information are fused to obtain the target color information; If the color difference information is greater than the second color threshold, and the region type is the target type, the target color information is determined based on the local color information.

6. An image processing apparatus, characterized in that, include: The image segmentation module is used to acquire the image to be processed and segment the image to be processed into multiple image blocks; The color temperature acquisition module is used to acquire the color temperature information of each image block and determine the local color temperature information of each image block through the color sensor array; The matrix determination module is used to determine the target color temperature information of each image block based on the color temperature information and the local color temperature information, and to obtain the color correction gain matrix of the target color temperature information; The color correction module is used to process the color information of the image block according to the color correction gain matrix, obtain the target color information of the image block, and generate the target image corresponding to the image to be processed; The determination of the target color temperature information for each image block based on the color temperature information and the local color temperature information includes: The color temperature information is compared with the local color temperature information to determine the difference information; If the difference information is less than the first threshold, the color temperature information is determined as the target color temperature information for each image block; if the difference information is greater than the first threshold and less than the second threshold, the local color temperature information and the color temperature information are weighted and fused according to the weight parameters to obtain the target color temperature information; if the difference information is greater than the second threshold, the local color temperature information is determined as the target color temperature information. The process of obtaining the color correction gain matrix of the target color temperature information includes: Interpolate the gain matrix of the color temperature information of each image block and the gain matrix of the local color temperature information to obtain the color correction gain matrix of the target color temperature information corresponding to each image block.

7. An electronic device, characterized in that, include: Imaging module, including color sensor array; processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the image processing method according to any one of claims 1-5 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image processing method according to any one of claims 1-5.

Images