Image processing method and device, electronic equipment, and storage medium

Abstract

The embodiment of the present disclosure relates to an image processing method and device, electronic equipment and storage medium, and relates to the technical field of images. The image processing method comprises: acquiring a to-be-processed image, and dividing the to-be-processed image to obtain a plurality of image blocks; determining target color temperature information of each image block according to color temperature information of each image block and local color temperature information of each image block determined by a color sensor array; acquiring region color temperature information of a target region in the to-be-processed image according to the target color temperature information, and acquiring a color correction gain matrix of the target region based on the region color temperature information; correcting color information of the target region according to the color correction gain matrix of the target region, acquiring target color information of the target region, and generating a target image corresponding to the to-be-processed image. The technical solution in the embodiment of the present disclosure can improve the accuracy of color correction.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image technology, and in particular, to an image processing method and device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] In the image processing process, color correction may be needed to improve the image quality.

[0003] In the related art, color correction uses color cards to calibrate under different color temperature light source conditions, and then determines the current scene color temperature in real time through a device, and finally calculates the color correction matrix needed to be used by interpolation. The above-mentioned method has certain limitations and can only be corrected as a whole, resulting in inaccurate color correction matrix and affecting the image quality.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the present disclosure is to provide an image processing method and device, an electronic device, and a computer readable storage medium, thereby at least partially overcoming the problem of poor image correction accuracy caused by the limitations and defects of the related art.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, an image processing method is provided, comprising: acquiring a to-be-processed image, and dividing the to-be-processed image to obtain a plurality of image blocks; determining target color temperature information of each image block according to color temperature information of each image block and local color temperature information of each image block determined by a color sensor array; acquiring region color temperature information of a target region in the to-be-processed image according to the target color temperature information, and acquiring a color correction gain matrix of the target region based on the region color temperature information; correcting color information of the target region according to the color correction gain matrix of the target region, acquiring target color information of the target region, and generating a target image corresponding to the to-be-processed image.

[0008] According to a second aspect of this disclosure, an image processing apparatus is provided, comprising: an image segmentation module for acquiring an image to be processed and segmenting the image to be processed into multiple image blocks; a color temperature acquisition module for determining target color temperature information of each image block based on color temperature information of each image block and local color temperature information of each image block determined by a color sensor array; a gain matrix determination module for acquiring regional color temperature information of a target region in the image to be processed based on the target color temperature information and acquiring a color correction gain matrix of the target region based on the regional color temperature information; and a color correction module for correcting the color information of the target region based on the color correction gain matrix of the target region, acquiring target color information of the target region, and generating a target image corresponding to the image to be processed.

[0009] According to a third aspect of this disclosure, an electronic device is provided, comprising: an image module including a color sensor array; a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image processing method of the first aspect and possible implementations thereof by executing the executable instructions.

[0010] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the image processing method of the first aspect described above and its possible implementations.

[0011] In the technical solution provided in this disclosure, on the one hand, the target color temperature information of each image block is obtained by acquiring local color temperature information through a color sensor array, thereby determining the color correction gain matrix corresponding to the regional color temperature information of the target area. This avoids the limitation of only being able to perform overall correction, and enables individual control of local image blocks represented by the target area, improving the independence and flexibility of image processing, as well as the targeting of image processing, increasing the application scope and improving application convenience. On the other hand, by calculating the color correction gain matrix in the target area through the target color temperature information of each image block, each image block in the target area can be independently controlled, and color correction can be performed through the color correction gain matrix of the target area, improving accuracy, color correction effect, and image quality.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0014] Figure 1 A schematic diagram illustrates an application scenario where the image processing method of the present disclosure embodiments can be applied.

[0015] Figure 2 The diagram illustrates an image processing method according to an embodiment of the present disclosure.

[0016] Figure 3 A schematic diagram of an image block is shown in an embodiment of this disclosure.

[0017] Figure 4 The schematic diagram illustrates the process of determining target color temperature information in different ways in the embodiments of this disclosure.

[0018] Figure 5 This diagram illustrates the weighting variation of target color temperature information in an embodiment of the present disclosure.

[0019] Figure 6 This diagram illustrates the determination of a target region in an embodiment of the present disclosure.

[0020] Figure 7 The diagram illustrates the color temperature transition in different regions in an embodiment of this disclosure.

[0021] Figure 8 A schematic diagram of radial transitions in an embodiment of this disclosure is shown.

[0022] Figure 9 The diagram illustrates the color transitions in different areas in an embodiment of this disclosure.

[0023] Figure 10 The schematic diagram illustrates the structure of the image signal processor in an embodiment of this disclosure.

[0024] Figure 11 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 12 A block diagram of an electronic device according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0027] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] This disclosure provides an image processing method that can be applied to scenarios where images are processed during the photography process. Figure 1 A schematic diagram of a system architecture for an image processing method and apparatus applicable to embodiments of the present disclosure is shown.

[0029] like Figure 1 As shown, terminal 101 can be a smart device with image processing capabilities, such as a smartphone, computer, tablet, smart speaker, smartwatch, in-vehicle device, wearable device, monitoring device, etc. The terminal may include a camera; the type of camera can be any type, as long as it can perform image processing. The number of cameras can be at least one, for example, one, four, etc., as long as they can take pictures. The image to be processed can be a captured image or each frame of a captured video.

[0030] In this embodiment, terminal 101 may include memory 102 and processor 103. The memory stores images, and the processor processes the images, such as performing white balance processing. Memory 102 may store an image 104 to be processed. Terminal 101 retrieves the image 104 to be processed from memory 102 and sends it to processor 103. Processor 103 divides the image to be processed into multiple image blocks, determines the local color temperature information and color temperature information of the image to be processed. Based on the color temperature information and the local color temperature information of each image block obtained by the color sensor array, the target color temperature information of each image block is determined. Furthermore, a color correction gain matrix for the target color temperature information in the target area is determined, so that the target area of ​​the image to be processed is color corrected using the color correction gain matrix, thereby generating a target image 105.

[0031] It should be noted that the image processing method provided in this embodiment can be executed by terminal 101. Alternatively, the image processing method can be configured within the terminal.

[0032] Next, refer to Figure 2 The image processing methods in the embodiments of this disclosure will be described in detail.

[0033] In step S210, the image to be processed is acquired, and the image to be processed is divided into multiple image blocks.

[0034] In this embodiment, the image to be processed can be an image captured by the camera module of the terminal, or it can be each frame of a captured video. The image to be processed can also be an image or each frame of a video directly obtained from a photo album or other storage location. The terminal can be any of the following: a smartphone, digital camera, smartwatch, wearable device, in-vehicle device, or surveillance camera, as long as it can capture images of the object and perform image processing. A smartphone is used as an example here. The camera module can include at least one camera, such as a main camera, telephoto camera, wide-angle camera, macro camera, or a combination thereof. The image to be processed can be various types of images, such as moving images or still images, etc.

[0035] After acquiring the image to be processed, it can be divided into multiple image blocks. Each image block can be a portion of the image to be processed, and the blocks do not overlap. Each image block can be the same size, and the number of image blocks can be determined based on the number of grid cells. For example, a grid region can be provided and applied to the image to be processed to divide it into multiple image blocks according to the grid region, with each image block corresponding one-to-one with a grid region. (See reference...) Figure 3As shown, a grid can contain multiple image blocks 301. For example, grid 00 corresponds to image block 00, grid 01 corresponds to image block 01, and so on. The size of the grid area can be set according to actual needs and hardware structure, that is, set according to actual needs within the limits of the hardware structure. For example, it can be row-based and column-based. Based on this, the color sensor can be associated with image blocks and grids, and the image to be processed can be divided into row × column image blocks, with each image block corresponding to each grid. Under the same field of view, the multi-window spatial range represented by the grid area is consistent with the image to be processed, so that the grid area can cover the image to be processed. Each grid can correspond to one window, hence it can be called a multi-window system.

[0036] In this embodiment, a color sensor array may be included, comprising multiple color sensors 302 arranged in an array. The number of color sensors can be determined based on the number of image patches. Furthermore, the color sensor array can be combined with multi-window information, thus each color sensor can be a multi-window color sensor. Each grid represents one color sensor, and each color sensor corresponds to each image patch. Since the image to be processed is divided into multiple grids, a row×col color sensor array is formed in space, which constitutes a multi-window color sensor array.

[0037] The multi-window color sensor array is an independent sensor that can be positioned to one side of any camera in the camera module, close to the camera module. The camera module can be a rear-facing camera module, and it can include at least one camera, such as a main camera, telephoto camera, wide-angle camera, macro camera, or a combination thereof. The specific placement and arrangement order of the multiple cameras can be determined according to actual needs and are not specifically limited here. For example, the color sensor array can be positioned to the left of the telephoto camera, to the right of the main camera, or below the last camera in the at least one camera setup, etc. The camera module and the color sensor array can be placed adjacent to each other or separated by a certain distance. The specific position of the color sensor array can be determined based on the calibration results during the actual application process or according to actual needs; it is not specifically limited here.

[0038] It should be noted that each module in the image signal processor can divide the image to be processed into blocks, and the resulting image blocks can be consistent with those obtained by dividing the image according to the grid, to ensure consistency and accuracy. For example, the color correction module can divide the image into multiple window regions to obtain the first image block; the 2D / 3D lookup table module can also divide the image into multiple window regions to obtain the second image block. The image division method is the same between different modules, and the first and second image blocks are basically consistent with the image blocks divided in step S210, to ensure consistency and accuracy between image blocks.

[0039] The multi-window area information of the color sensor array is designed to correspond with the color correction module. The window size of the color correction module should be equal to, slightly larger than, or smaller than the multi-window area of ​​the color sensor array, that is, the number of rows and columns of the two should be the same.

[0040] Next, continue to refer to Figure 2 As shown, in step S220, the target color temperature information of each image block is determined based on the color temperature information of each image block and the local color temperature information of each image block determined by the color sensor array.

[0041] In this embodiment, the color temperature information of each image block can first be obtained through an image signal processor (ISP), specifically represented by cct1. In addition, the color temperature of a local area of ​​the current scene can be calculated through a color sensor array, i.e., the local color temperature information of each image block can be obtained, represented by cct2. The color sensor can be a sensor used to detect scene color, color temperature, spectrum, and other related information, and can be used to detect the color information of the object corresponding to each image block and the color temperature information of the current scene. The object can be any type of object contained in each image block, such as a person, etc.

[0042] In this embodiment of the disclosure, the target color temperature information of an image patch can be determined based on both color temperature information and local color temperature information. For example, the color temperature information can be compared with the local color temperature information to determine the difference information; based on the comparison result of the difference information and a threshold parameter, different methods can be selected to obtain the target color temperature information. The threshold parameter may include a first threshold and a second threshold, and the first threshold Th1 is less than the second threshold Th2.

[0043] Figure 4 The diagram illustrates a flowchart of different methods for determining target color temperature information. (See reference) Figure 4 As shown, the main steps include:

[0044] In step S410, it is determined whether the difference information is less than the first threshold; if yes, proceed to step S420; if no, proceed to step S430.

[0045] In step S420, if the difference information is less than the first threshold, the target color temperature information is determined based on the color temperature information;

[0046] In step S430, it is determined whether the difference information is less than the second threshold; if yes, proceed to step S440; if no, proceed to step S450.

[0047] In step S440, if the difference information is greater than a first threshold and less than a second threshold, the color temperature information is adjusted using local color temperature information to determine the target color temperature information;

[0048] In step S450, if the difference information is greater than the second threshold, the target color temperature information is determined based on the local color temperature information.

[0049] In this embodiment of the disclosure, as the difference information changes, the weight of the target color temperature information in the total color temperature information gradually decreases, as detailed in the following reference. Figure 5 The weighting change diagram is shown below. When the difference information is greater than the first threshold, the target color temperature information has a weight of 1 in the total color temperature information. When the difference information is between the first and second thresholds, the weight of the target color temperature information gradually decreases. When the difference information is greater than the second threshold, the weight of the target color temperature information is 0, that is, the target color temperature information has a weight of 1 in the local color temperature information acquired by the color sensor array.

[0050] Based on this, if the difference between the two is less than the first threshold, the color sensor array does not need to correct the color temperature information obtained by the image signal processor, and therefore directly uses the color temperature information as the target color temperature information for each image block.

[0051] If the difference between the two is greater than the first threshold and less than the second threshold, the local color temperature information obtained by the color sensor array needs to be adjusted to adjust the color temperature information obtained by the image signal processor. Specifically, the target color temperature information of each image block can be determined jointly by the color temperature information obtained by the color sensor array and the image signal processor. That is, the color temperature information is interpolated by the local color temperature information obtained by the color sensor array. For example, the target color temperature information of each image block can be obtained by interpolating all color temperature information calculated in different ways. Interpolation can be a weighted summation operation. For example, the target color temperature information can be obtained by weighted fusion of the local color temperature information obtained by the color sensor array and the color temperature information according to the corresponding weight parameters, as shown in formula (1):

[0052] cct=cct1*w1+cct2*w2 formula (1)

[0053] Here, cct1 represents the color temperature information acquired by the image signal processor, and cct2 represents the local color temperature information of each image block acquired by the color sensor array.

[0054] Adjusting the color temperature information by using local color temperature information obtained from a color sensor array can improve the accuracy of the target color temperature information for each image block, and also avoids the influence of the environment on the color temperature in related technologies, thus improving stability.

[0055] If the difference between the two is greater than the second threshold, the target color temperature information of each image block can be determined based on the local color temperature information obtained by the color sensor array.

[0056] After determining the target color temperature information for each image block, the target color temperature information of each image block can be smoothed to achieve a smooth transition between the target color temperature information of all image blocks, reducing the abrupt effect caused by the different target color temperature information between the local blocks represented by each image block. This smoothing process can be achieved through low-pass filtering.

[0057] Next, in step S230, the regional color temperature information of the target area in the image to be processed is obtained according to the target color temperature information, and the color correction gain matrix of the target area is obtained.

[0058] In this embodiment of the disclosure, a color correction gain matrix is ​​used to implement color correction. Color correction refers to correcting the difference between the current color and the target color of an image using a 24-color chart. The color correction gain matrix is ​​used to correct the color parameters of each pixel to adjust them to the target color parameters. The color parameters can be RGB values.

[0059] To address the limitation of targeted processing of specific regions in related technologies, mask information can be provided to select a target region from the image to be processed. At least one mask can be used, determined based on specific needs; this example uses one mask. The target region can be a portion of the image to be processed that matches the mask information. (Reference) Figure 6 As shown, the target region 602 can be selected from the image 600 to be processed using mask information 601, and other regions besides the target region can be defined as reference regions 603. The target region can be a region that requires special attention or a region that requires fine-grained adjustment, such as a face region or a region with many details. The target region can contain at least one image patch, and the image patch can be a complete image patch or a partial image patch, depending on the size of the mask information.

[0060] After acquiring the target region, the regional color temperature information (cct4) of the target region can be output based on the target color temperature information of the image blocks contained within it. For example, the regional color temperature information can be the target color temperature information for each image block individually, such as the target color temperature information for image block 1, image block 2, and image block 3; or it can be the color temperature information obtained by integrating the target color temperature information of multiple image blocks. Here, we will illustrate this by taking the regional color temperature information as the target color temperature information for each image block individually. Simultaneously, the color temperature information of the reference region in the image to be processed, excluding the target region, can be determined as the third color temperature information (cct3). The third color temperature information is different from the fourth color temperature information.

[0061] After obtaining the regional color temperature information of the target area, the regional color temperature information of the target area can be adjusted according to the first adjustment parameter to output the fourth color temperature information, while keeping the color temperature information of the reference area unchanged to obtain the third color temperature information. For example, the first adjustment parameter is determined according to actual needs, such as user requirements or system settings. The first adjustment parameter may include the image blocks to be adjusted and the degree of adjustment. During adjustment, all or part of the image blocks to be adjusted and the adjustment range can be used to adjust the regional color temperature information of the target area to output the fourth color temperature information, achieving precise and independent control of the fourth color temperature information of the target area and improving flexibility. It should be noted that for each image block contained in the target area, since the proportion of each image block in the target area is different, the adjustment priority of each image block can be different. Specifically, the adjustment priority can be positively correlated with the proportion of the image block, that is, the larger the proportion of the image block in the target area, the higher the adjustment priority. The adjustment direction of different image blocks may be different, and is determined according to actual needs.

[0062] To avoid differences in color temperature information between different regions, the fourth color temperature information within the target area and the third color temperature information of the reference area outside the target area can be smoothly transitioned. This smooth transition can be radial or other methods; a radial transition will be used as an example here.

[0063] In this embodiment, the central region of the third color temperature information of the reference region outside the target region is taken as the radial origin, and a radial transition is made from the fourth color temperature information of the target region to the third color temperature information of the reference region. Figure 7 As shown in the image. The transition between the two can be a series of weight transition methods, such as a smooth curve weight radial distribution or a linear distribution, as shown in the reference. Figure 8As shown, no specific limitations are made here. A smooth curve weight radial distribution means the weights of the transition method can be distributed along a smooth curve, while a linear distribution means the weights of the transition method are distributed linearly. When it is a linear distribution, the fourth color temperature information can be used when the radial distance is less than the first threshold Th1; when the radial distance is greater than the first threshold Th1 but less than the first threshold Th2, the fourth color temperature information and the third color temperature information are weighted and fused; and when the radial distance is greater than the second threshold Th2, the third color temperature information is used. During weighted fusion, the weight of the fourth color temperature information decreases as the radial distance increases, gradually transitioning to the third color temperature information.

[0064] In this embodiment of the disclosure, the target area is obtained through mask information, and then the regional color temperature information of the target area can be obtained. Furthermore, the color correction gain matrix of the target area can be determined based on the regional color temperature information.

[0065] Specifically, the color correction gain matrix can be determined based on the regional color temperature information within the target area and the calibration light source results of the image signal processor. The regional color temperature information within the target area can be the target color temperature information of each image block contained within the target area, and the calibration light source results refer to the color temperature information acquired by the image signal processor. In some embodiments, the calibrated color temperature information of each image block has a corresponding gain matrix, and the gain matrices of each image block can be the same or different. Furthermore, the local color temperature information of each image block acquired by the color sensor array also has a corresponding gain matrix. The gain matrices for both the calibrated color temperature information and the local color temperature information can, for example, be 3×3 matrices.

[0066] Based on this, the gain matrix of the color temperature information of each image block and the gain matrix of the local color temperature information can be interpolated to obtain the color correction gain matrix of the target color temperature information corresponding to each image block, and the color correction gain matrix of all image blocks included in the target region is determined as the color correction gain matrix of the target region. For example, the gain matrix of the color temperature information and the gain matrix of the local color temperature information can be fused according to corresponding weight parameters. This fusion can be achieved through a weighted summation operation. The weight parameters are negatively correlated with the distance from the corresponding color temperature information to the target color temperature information; that is, the greater the distance from the corresponding color temperature information to the target color temperature information, the smaller the weight parameter. The weight parameter of the gain matrix of each color temperature information can be determined based on the distance from another color temperature information to the target color temperature information. For example, point A represents color temperature information, point B represents local color temperature information, point C represents target color temperature information, and the distance between A and C is m1, and the distance between B and C is m2. The color temperature information at point A is 2000K, the gain matrix is ​​A1 and its weight parameter is m2 / m1+m2, the local color temperature information at point B is 3000K and the gain matrix is ​​B1 and its weight parameter is m1 / m1+m2, then the color correction gain matrix of the target color temperature information at point C can be expressed as m2 / m1+m2*A1+m1 / m1+m2*B1.

[0067] After obtaining the color correction gain matrix for each image patch, it can be smoothed again. Smoothing can be done through low-pass filtering, resulting in the color correction gain matrix for each image patch. The color correction gain matrix can be a red-green-blue gain matrix. Based on this, the color correction gain matrix of the target region can be obtained from the color correction gain matrices of all image patches contained within the target region; that is, the color correction gain matrix of the target region is the color correction gain matrix of all image patches contained within the target region. As the fourth color temperature information within the target region is adjusted, the color correction gain matrix of the target region can also change, depending on the adjustment parameters. However, the color correction gain matrix of the reference region remains constant.

[0068] Continue to refer to Figure 2 As shown, in step S240, the color information of the target region is corrected according to the color correction gain matrix of the target region to generate the target image corresponding to the image to be processed.

[0069] Furthermore, the color correction gain matrix can be applied to the target region of the image to be processed. For example, image blocks can be obtained from the target region, and the color information of each image block can be corrected according to the color correction gain matrix of each image block. Specifically, the color correction gain matrix of each image block contained in the target region can be multiplied with the color parameters of the pixels of each image block to adjust the color parameters of the pixels of each image block in the target region to the required target color information, making the image color more consistent with actual needs and improving image quality.

[0070] In this embodiment of the disclosure, by calculating the color correction gain matrix of each image block and obtaining the color correction gain matrix of the target region, it is possible to independently control a portion of the image to be processed, avoiding the limitation of only being able to process the whole in related technologies, improving the flexibility and targeting of image processing, and also improving image quality.

[0071] In some scenarios, the overall colors of the raw images captured by the camera are not vibrant enough, but users also want to capture images of different scenes, such as green plants, grass, flowers, blue skies, white clouds, beaches, buildings, and animals. Therefore, specific transformations can be made to the hue and saturation of certain colors. For example, when the scene is green plants, a deeper green is desired while other colors remain unchanged. This can be achieved through color mapping using a 2D / 3D LUT. Color mapping refers to mapping the original image or video image to colors based on a pre-adjusted color mapping table. The color mapping table is a three-dimensional lookup table; the input consists of three color components (IN_RGB), and the lookup table directly returns the corresponding output color components (OUT_RGB). In this embodiment, the color sensor array can also be associated with a 2D / 3D LUT (2D / 3D lookup table) to enhance the capabilities of the 2D / 3D LUT. It should be noted that the window size of the 2D / 3D lookup table module should be equal to, larger than, or smaller than the multi-window area of ​​the color sensor. Furthermore, the method of dividing the image blocks and the number of resulting image blocks are essentially the same.

[0072] Based on this, the method further includes: acquiring target color information of the target region in the image to be processed; and mapping the target color information using the color difference information of the target region and the mapping method corresponding to the region type. The target region can be a region requiring special attention, such as a face, tree area, etc. The target region here can be the same as or different from the target region of the aforementioned color correction module, depending on the actual processing requirements. It should be noted that the mapping can be performed on the target color information or on the original color information in the image to be processed; this is not limited here. That is, the 2D / 3D LUT module can be executed simultaneously with the color correction module or after the color correction module.

[0073] In some embodiments, the target color information of the image block in the target region can be color-mapped by combining the local color information obtained by the color sensor array and the reference color information obtained by the lookup table module, so as to adjust the target color information to the required corrected color information. Alternatively, the original color information of the image block in the target region can also be mapped. Color mapping can be used to implement color correction to adjust color parameters, such as adjusting the values ​​of the three RGB sub-pixels. In this embodiment of the present disclosure, color mapping can be performed by combining the parameter mapping relationship of the color sensor array and the lookup table module itself. Specifically, color mapping by combining the local color information obtained by the color sensor array and the reference color information obtained by the lookup table module can be performed by combining a color threshold parameter and the region type of the target region to determine the mapping method. The color threshold parameter can be a first color threshold and a second color threshold.

[0074] For example, the local color information can be compared with the reference color information to determine the color difference information; based on the comparison result of the color difference information and the color threshold parameter, the mapping method corresponding to the comparison result can be selected to obtain the corrected color information.

[0075] In this embodiment of the disclosure, when determining that the region type of the image block contained in the target region is the target type, the target color information can be determined by combining the comparison results between color difference information and color threshold parameters, and by combining the color threshold parameters and the region type, using different mapping methods composed of local color information and reference color information.

[0076] In some embodiments, it is first determined whether the region type of the image patch contained in the target region is the target type. The target type refers to the type of region whose color needs to be adjusted, which can be determined based on the type of object contained in the image patch, the color information of the image patch, or whether a user operation has been received. For example, if the type of object contained in the image patch is human face skin color, blue sky, or green plants, then the region type of the image patch can be determined to be the target type.

[0077] If the color difference information is less than a first color threshold, the target color information or color information of the image block is mapped by obtaining the parameter mapping relationship of the reference color information to obtain the corrected color information. Alternatively, the reference color information of the image block can be directly determined as the corrected color information. In this step, the parameter mapping relationship can be a 2D / 3D lookup table, such as a color mapping table. For example, the target color information of the image block contained in the target region can be input into the color mapping table, and the corrected color information corresponding to the color mapping table can be output. The intensity of color enhancement can be the intensity set by the lookup table itself, or it can be adjusted according to actual needs. For example, the current RGB three color components of a pixel in the image block can be input into the color mapping table, and the corresponding RGB three color mapping components are output through the color mapping table. When performing color mapping through the mapping table, since the image block can contain multiple pixels, each pixel can be adjusted according to the corresponding color mapping table to obtain the corrected color information output after color adjustment.

[0078] If the color difference information is greater than a first color threshold and less than a second color threshold, the corrected color information after color mapping of the target color information is determined by fusing local color information and reference color information. In this step, the local color information and reference color information can be weighted and fused according to their respective weights to obtain the corrected color information. As the color difference information increases, the weight of the local color information gradually increases, while the weight of the reference color information gradually decreases.

[0079] If the color difference information is greater than the second color threshold, the target color information is determined based on the local color information. Specifically, if the color difference information is greater than the second color threshold, it indicates that the mapped color information may be inaccurate. In this case, the local color information output by the color sensor array needs to be used as the correction color information, meaning the local color information has a higher weight.

[0080] Furthermore, the corrected color information obtained after color mapping for each image patch can be smoothly transitioned. This smooth transition can be achieved using low-pass filtering, resulting in a smoother transition between different image patches in the spatial domain. This avoids abrupt color changes between image patches, thus achieving a smooth transition and improving image quality.

[0081] After obtaining the corrected color information for each image block, the target color information of each image block can be mapped to corrected color information based on the comparison results between the color temperature and color difference information of the image blocks contained in the target region and the color threshold parameters. Specifically, the pixel information of each image block contained in the target region can be mapped. Based on this, the corrected color information in the target region can be determined as the first color information RGB1, and the corrected color information in the reference region can be determined as the second color information RGB2, with the first color information and the second color information being different. Furthermore, the corrected color information in the target region can be adjusted according to a second adjustment parameter to update the first color information while keeping the second color information in the reference region unchanged. The second adjustment parameter can include the image blocks to be adjusted and the degree of adjustment. During adjustment, all or part of the adjustment can be performed according to the image blocks to be adjusted and the adjustment range, thereby adjusting the first color information of the target region, achieving precise and independent control of the first color information of the target region, and improving flexibility. In this embodiment of the disclosure, to avoid abrupt changes between corrected color information in different regions, a radial transition is made from the center region of the first color information of the target region to the second color information of the reference region, with the center region of the first color information of the target region as the radial origin. Figure 9 As shown in the figure. The transition between the two can be a series of weight transition methods such as smooth curve weight radial distribution or linear distribution, which are not limited here.

[0082] Based on this, a target region can be determined through mask information, and then its mapping method can be determined according to the image blocks contained in the target region. The target color information of each image block in the target region can be mapped to obtain the corrected color information, which can provide a distinguishing dimension of image content, realize targeted processing of target regions and other parts, improve the richness and flexibility of color expression, and improve image quality.

[0083] Figure 10 A schematic diagram of the image signal processor is shown for reference. Figure 10 As shown, the image signal processor may include a color sensor array 1001, a grid 1002, an algorithm module 1003, and may also include modules such as a color correction module 1004, a tone mapping module 1005, a 2D / 3D LUT 1006, a segmentation module 1007, and a sensor 1008.

[0084] refer to Figure 10As shown, the grid information acquired by the color sensor array is associated with the grid acquired by the color correction module and the 2D / 3D LUT to ensure that the segmented image blocks are basically consistent. The local color temperature information output by the algorithm module from the color sensor array is input to the color correction module, and the local color information output by the algorithm module from the color sensor array is input to the 2D / 3D LUT. The color correction module and the 2D / 3D LUT are usually located in the second stage of the ISP processing flow, namely the RGB domain stage. The segmentation module is connected to the color correction module, the 2D / 3D LUT, and the sensor.

[0085] In this embodiment of the disclosure, by introducing a color sensor array to obtain local color temperature information and local color information, the color adjustment capability of the color correction module and the 2D / 3D LUT module can be improved, thereby improving the accuracy of the target color information.

[0086] In summary, the technical solution in this disclosure, after introducing a segmentation module, can obtain the target region in the image to be processed through mask information, and then perform local color correction and color mapping processing on the image content of the local region represented by the target region in the image to be processed. This avoids the limitation of only being able to process the whole in related technologies, and improves the flexibility and targeting of image processing; it can effectively improve the accuracy of color restoration of the image and improve the image quality of the target image.

[0087] This disclosure provides an image processing apparatus, with reference to... Figure 11 As shown, the image processing device 1100 may include:

[0088] Image segmentation module 1101 is used to acquire an image to be processed and segment the image to be processed into multiple image blocks;

[0089] The color temperature acquisition module 1102 is used to determine the target color temperature information of each image block based on the color temperature information of each image block and the local color temperature information of each image block determined by the color sensor array.

[0090] The gain matrix determination module 1103 is used to obtain the regional color temperature information of the target area in the image to be processed according to the target color temperature information, and to obtain the color correction gain matrix of the target area based on the regional color temperature information;

[0091] The color correction module 1104 is used to correct the color information of the target region according to the color correction gain matrix of the target region, obtain the target color information of the target region, and generate the target image corresponding to the image to be processed.

[0092] In one exemplary embodiment of this disclosure, the gain matrix determination module includes: a region color temperature acquisition module, configured to acquire a target region from the image to be processed based on mask information, and determine the region color temperature information of the target region based on the target color temperature information of the image blocks contained in the target region.

[0093] In one exemplary embodiment of this disclosure, the regional color temperature acquisition module includes: an adjustment module, configured to adjust the target color temperature information of the image blocks contained in the target region according to a first adjustment parameter, so as to adjust the regional color temperature information of the target region.

[0094] In one exemplary embodiment of this disclosure, the color temperature acquisition module includes: a difference calculation module, used to compare the color temperature information with the local color temperature information to determine the difference information; and a mode selection module, used to select different modes to acquire the target color temperature information based on the comparison result of the difference information and a threshold parameter.

[0095] In one exemplary embodiment of this disclosure, the gain matrix determination module includes: an interpolation module, configured to interpolate the color temperature information of each image block and the local color temperature information of each image block determined by the color sensor array, to determine the color correction gain matrix of the target color temperature information of each image block, so as to determine the color correction gain matrix within the target area.

[0096] In one exemplary embodiment of this disclosure, the apparatus further includes: a color mapping module, configured to determine a mapping method based on the color difference information of the target area and the area type, and to map the target color information into corrected color information according to the mapping method.

[0097] In an exemplary embodiment of this disclosure, the color mapping module is configured to: if the color difference information of an image patch in the target region is less than a first color threshold and the region type is a target type, determine the corrected color information according to the parameter mapping relationship for obtaining the reference color information; if the color difference information is greater than the first color threshold and less than a second color threshold, and the region type is a target type, fuse the local color information and the reference color information to obtain the corrected color information; if the color difference information is greater than the second color threshold and the region type is a target type, determine the corrected color information according to the local color information.

[0098] It should be noted that the specific details of each part of the above-mentioned image processing apparatus have been described in detail in the implementation of the image processing method section. For any undisclosed details, please refer to the implementation of the method section, and therefore will not be repeated here.

[0099] An exemplary embodiment of this disclosure also provides an electronic device. This electronic device may be the terminal 101 described above. Generally, the electronic device may include a processor and a memory, the memory being used to store executable instructions of the processor, the processor being configured to perform the image processing method described above by executing the executable instructions.

[0100] The following is based on Figure 12 Taking the mobile terminal 1200 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 12 The structure can also be applied to fixed types of equipment.

[0101] like Figure 12 As shown, the mobile terminal 1200 may specifically include: a processor 1201, a memory 1202, a bus 1203, a mobile communication module 1204, an antenna 1, a wireless communication module 1205, an antenna 2, a display screen 1206, a camera module 1207, an audio module 1208, a power module 1209, and a sensor module 1210.

[0102] Processor 1201 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The image denoising method in this exemplary embodiment can be executed by an AP, GPU, or DSP. When the method involves neural network-related processing, it can be executed by an NPU. For example, the NPU can load neural network parameters and execute neural network-related algorithm instructions.

[0103] An encoder encodes (compresses) images or videos to reduce data size for easier storage or transmission. A decoder decodes (decompresses) the encoded data to restore the original image or video data. The mobile terminal 1200 can support one or more encoders and decoders, such as image formats like JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), and BMP (Bitmap), and video formats like MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, and HEVC (High Efficiency Video Coding).

[0104] The processor 1201 can be connected to the memory 1202 or other components via the bus 1203.

[0105] The memory 1202 can be used to store computer executable program code, which includes instructions. The processor 1201 executes various functional applications and data processing of the mobile terminal 1200 by running the instructions stored in the memory 1202. The memory 1202 can also store application data, such as images, videos, and other files.

[0106] The communication function of mobile terminal 1200 can be implemented through mobile communication module 1204, antenna 1, wireless communication module 1205, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1204 can provide 3G, 4G, 5G and other mobile communication solutions for mobile terminal 1200. Wireless communication module 1205 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1200.

[0107] The display screen 1206 is used to implement display functions, such as displaying user interfaces, images, and videos. The camera module 1207 is used to implement shooting functions, such as capturing images and videos, and may include a color sensor array. The audio module 1208 is used to implement audio functions, such as playing audio and capturing voice. The power module 1209 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring battery status. The sensor module 1210 may include one or more sensors to implement corresponding sensing and detection functions. For example, the sensor module 1210 may include an inertial sensor, which is used to detect the motion posture of the mobile terminal 1200 and output inertial sensing data.

[0108] It should be noted that the present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist alone and not assembled into the electronic device.

[0109] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0110] A computer-readable storage medium can be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0111] A computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments.

[0112] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0113] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0114] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0115] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims. It should be understood that this disclosure is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: The image to be processed is acquired, and the image to be processed is divided into multiple image blocks; The target color temperature information of each image block is determined based on the color temperature information of each image block and the local color temperature information of each image block determined by the color sensor array. Based on the target color temperature information, obtain the regional color temperature information of the target region in the image to be processed, and obtain the color correction gain matrix of the target region based on the regional color temperature information; The color information of the target region is corrected according to the color correction gain matrix of the target region to obtain the target color information of the target region, so as to generate the target image corresponding to the image to be processed; The method further includes: determining a mapping method based on the color difference information and region type of the target region, and mapping the target color information to corrected color information according to the mapping method; wherein, If the color difference information of the image block in the target region is less than the first color threshold and the region type is the target type, the corrected color information is determined according to the parameter mapping relationship of the obtained reference color information; If the color difference information is greater than the first color threshold and less than the second color threshold, and the region type is the target type, the local color information and the reference color information are fused to obtain the corrected color information; If the color difference information is greater than the second color threshold, and the region type is the target type, the corrected color information is determined based on the local color information.

2. The image processing method according to claim 1, characterized in that, The step of obtaining the regional color temperature information of the target region in the image to be processed based on the target color temperature information includes: The target region is obtained from the image to be processed based on the mask information, and the region color temperature information of the target region is determined based on the target color temperature information of the image blocks contained in the target region.

3. The image processing method according to claim 2, characterized in that, Determining the region color temperature information of the target region based on the target color temperature information of the image blocks contained in the target region includes: The target color temperature information of the image blocks contained in the target region is adjusted according to the first adjustment parameter, so as to adjust the regional color temperature information of the target region.

4. The image processing method according to claim 1, characterized in that, The step of determining the target color temperature information of each image block based on the color temperature information of each image block and the local color temperature information of each image block determined by the color sensor array includes: The color temperature information is compared with the local color temperature information to determine the difference information; Based on the comparison results between the difference information and the threshold parameter, different methods are selected to obtain the target color temperature information.

5. The image processing method according to claim 1, characterized in that, The step of obtaining the color correction gain matrix of the target region based on regional color temperature information includes: The color temperature information of each image block and the local color temperature information of each image block determined by the color sensor array are interpolated to determine the color correction gain matrix of the target color temperature information of each image block, so as to determine the color correction gain matrix in the target area.

6. An image processing apparatus, characterized in that, include: The image segmentation module is used to acquire the image to be processed and segment the image to be processed into multiple image blocks; The color temperature acquisition module is used to determine the target color temperature information of each image block based on the color temperature information of each image block and the local color temperature information of each image block determined by the color sensor array. The gain matrix determination module is used to obtain the regional color temperature information of the target region in the image to be processed based on the target color temperature information, and to obtain the color correction gain matrix of the target region based on the regional color temperature information. The color correction module is used to correct the color information of the target region according to the color correction gain matrix of the target region, obtain the target color information of the target region, and generate the target image corresponding to the image to be processed. The color mapping module is used to determine a mapping method based on the color difference information and region type of the target region, and to map the target color information into corrected color information according to the mapping method. Specifically, if the color difference information of an image patch in the target region is less than a first color threshold and the region type is a target type, the corrected color information is determined based on the parameter mapping relationship for obtaining reference color information. If the color difference information is greater than the first color threshold and less than a second color threshold, and the region type is a target type, the local color information and reference color information are fused to obtain the corrected color information. If the color difference information is greater than the second color threshold and the region type is a target type, the corrected color information is determined based on the local color information.

7. An electronic device, characterized in that, include: Imaging module, including color sensor array; processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the image processing method according to any one of claims 1-5 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image processing method according to any one of claims 1-5.

Images