Image processing method, device, electronic device and readable storage medium

Abstract

The present application discloses an image processing method, apparatus, electronic device, and readable storage medium, belonging to the field of image processing technology. The method is performed by an electronic device, the electronic device including a first lens module and a second lens module, wherein the field of view of the first lens module is smaller than the field of view of the second lens module; the method comprises: when the electronic device displays an image captured by the first lens module, obtaining first color statistical information of a target lens module of the electronic device at a first moment, wherein the target lens module includes the second lens module; determining first color compensation information mapped to the first color statistical information based on a first color mapping relationship; and performing white balance processing on a first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] In related technologies, electronic devices can integrate multiple lens modules, such as ultra-wide-angle lens modules, wide-angle lens modules, telephoto lens modules, etc., to meet different shooting requirements.

[0003] Different lens modules have different fields of view (FOVs). For lens modules with small FOVs, such as telephoto lens modules, the limited FOV can easily lead to color casts in the Auto White Balance (AWB) due to insufficient grayscale in the color statistics, resulting in poor white balance processing. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and readable storage medium, which can solve the problem of poor white balance processing effect.

[0005] In a first aspect, the embodiments of the present application provide an image processing method.

[0006] The method is performed by an electronic device, the electronic device comprising a first lens module and a second lens module, wherein the field of view of the first lens module is smaller than the field of view of the second lens module; the method comprising:

[0007] When the electronic device displays an image captured by the first lens module, obtaining first color statistical information of a target lens module of the electronic device at a first moment, wherein the target lens module includes the second lens module;

[0008] Determining first color compensation information mapped to the first color statistical information based on a first color mapping relationship, wherein the first color mapping relationship is a color mapping relationship of the electronic device, the color mapping relationship being a mapping relationship between the color statistical information of the target lens module and the color compensation information of the first lens module; and the first color compensation information being color compensation information of the first lens module of the electronic device at the first moment;

[0009] Based on the first color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, applied to an electronic device, wherein the electronic device includes a first lens module and a second lens module, wherein the field of view of the first lens module is smaller than the field of view of the second lens module; the method includes:

[0011] a first acquisition module, configured to acquire, when the electronic device displays an image captured by the first lens module, first color statistical information of a target lens module of the electronic device at a first moment, wherein the target lens module includes the second lens module;

[0012] A first determining module is configured to determine first color compensation information mapped to the first color statistical information based on a first color mapping relationship, wherein the first color mapping relationship is a color mapping relationship of the electronic device, the color mapping relationship is a mapping relationship between the color statistical information of the target lens module and the color compensation information of the first lens module; and the first color compensation information is color compensation information of the first lens module of the electronic device at the first moment;

[0013] The first white balance processing module is configured to perform white balance processing on a first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information.

[0014] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0018] In an embodiment of the present application, when an electronic device displays an image captured by a first lens module with a small FOV, first color statistical information of a target lens module with a large FOV at a first moment can be obtained, and then, based on a mapping relationship between the color statistical information of the target lens module of the electronic device and the color compensation information of the first lens module, first color compensation information corresponding to the first color statistical information is determined, that is, the color compensation information of the first lens module at the first moment is obtained. Thereafter, white balance processing is performed on the first image captured by the first camera module of the electronic device at the first moment based on the first color compensation information. In an embodiment of the present application, through a mapping relationship between the color statistical information of the target lens module of the electronic device and the color compensation information of the first lens module, the color statistical information of the target lens module with a large FOV can be used to accurately determine the color compensation information of the first lens module with a small FOV, thereby accurately color compensating the lens module with a small FOV using the color statistical information of the target lens module with a large FOV. Since the target lens module with a large FOV can obtain more color statistical information, the white balance processing effect of the lens module with a small FOV can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1a This is one of the schematic diagrams of color mapping provided in the embodiments of the present application;

[0020] Figure 1b Schematic diagram of the relationship between the light source and AWB gain provided in an embodiment of the present application;

[0021] Figure 2 is a flowchart of the image processing method provided in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of image selection for color mapping provided in an embodiment of the present application;

[0023] Figure 4a This is the second schematic diagram of color mapping provided in an embodiment of the present application;

[0024] Figure 4b This is the third schematic diagram of color mapping provided in the embodiment of the present application;

[0025] Figure 4c This is a schematic diagram of gray area selection provided in an embodiment of the present application;

[0026] Figure 5 Schematic diagram of obtaining the calibration coefficient provided in the embodiment of the present application;

[0027] Figure 6 This is a schematic diagram of obtaining the prediction function provided in the embodiment of the present application;

[0028] Figure 7Schematic diagram of obtaining AWB Gain provided in an embodiment of the present application;

[0029] Figure 8a This is a schematic diagram of the color gamut before two-dimensional space color mapping provided by an embodiment of the present application;

[0030] Figure 8b This is a schematic diagram of the color gamut after two-dimensional space color mapping provided by an embodiment of the present application;

[0031] Figure 9 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0032] Figure 10 This is one of the structural diagrams of the electronic device provided in the embodiment of the present application;

[0033] Figure 11 This is the second structural diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0035] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] For ease of understanding, some contents involved in the embodiments of this application are described below.

[0037] Multi-camera Synchronization System (MCSS): A hardware synchronization method based on the existing read time and superimposed on the unique low frame rate and low power consumption related characteristics of MCSS.

[0038] Spatial Alignment Transform (SAT): This feature enables automatic switching between different lens modules during preview or continuous zoom recording, reducing image misalignment and distortion caused by switching.

[0039] Color mapping: Based on the color information of the lens module before switching and combined with the algorithm model, the color information of the lens module after switching can be predicted.

[0040] White balance (Auto White Balance, AWB): refers to the process of measuring and adjusting white objects in the scene in a digital camera or mobile phone to ensure that the white part of the captured image achieves a true and natural color and that other colors are also accurately restored.

[0041] One-time programmable data or program (OTP): refers to the color-related characteristic parameters of the lens module, such as the gray point response results at different color temperatures.

[0042] Golden: Statistics of the entire batch of lens modules reveal that their OTP parameters show a normal distribution as the number of samples increases. In this embodiment of the application, the sample whose OTP parameters conform to the mean of the normal distribution is called Golden, indicating that its color characteristics best represent the statistical laws of the entire batch of lens modules.

[0043] With the development of mobile imaging across the entire mobile phone industry, mainstream manufacturers are no longer solely focused on piling up hardware specifications and functional parameters. Instead, they are increasingly focusing on fundamental user experience, such as ensuring consistent effects during SAT lens module switching and color accuracy for small FOV lens modules (such as telephoto lenses). These experience optimization tasks require engineers to devote more thought, breaking down technologies from different dimensions and designing and deploying solutions at the system level.

[0044] To address this issue, the color information of the lens module after the switch can be mapped based on the color information of the lens module before the switch, thereby improving color consistency before and after the lens module switch. At the same time, when the small FOV lens module is being displayed, the large FOV lens module can obtain more statistical information, assisting the small FOV lens module in real-time color calibration (AWB).

[0045] In some embodiments, in order to improve the color accuracy of the small FOV lens module, the large FOV lens module can be set to be always on and the statistical results can be mapped to the small FOV lens module. Figure 1aAs shown, for example, when zooming from the wide-angle lens module to the telephoto lens module: From 2.8x to 3.3x, both the wide-angle and telephoto lenses are enabled simultaneously to ensure color consistency before and after the lens module switch. However, after zooming from 3.3x, the wide-angle lens module (large FOV) remains enabled to ensure color accuracy in the telephoto lens module (small FOV). Therefore, this dual-lens module strategy increases the power consumption of the phone.

[0046] The main principle of the color mapping solution is to map the AWB gain (AWB Gain) based on the prior information of multiple lens modules. In theory, it is necessary to find a one-to-one mapping function among the light source, the AWB Gain of the large FOV lens module, and the AWB Gain of the small FOV lens module. However, since the integral results of the ambient reflection spectrum and the lens module response curve are the same color but different spectra, it cannot be guaranteed that the three can maintain a one-to-one mapping relationship under all reflection spectra. In other words, there is a probability that the AWB Gain of the large FOV lens module is different under different light sources, but after mapping, the AWB Gain results of the small FOV lens module are the same or very close, which will cause color cast of the small FOV lens module. For ease of understanding, please refer to Figure 1b .

[0047] Based on this, the present invention proposes a new color mapping solution by improving the MCSS hard synchronization system. This solution can significantly reduce power consumption while solving the color jump problem caused by lens module switching during the SAT process and improving the color accuracy of lens modules with small FOV.

[0048] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0049] The image processing method of the embodiment of the present application can be executed by an electronic device, or applied to an electronic device. The electronic device of the embodiment of the present application may include, but is not limited to, a first lens module and a second lens module, wherein the field of view of the first lens module is smaller than the field of view of the second lens module. Furthermore, the electronic device may also include a third lens module, the field of view of the third lens module is larger than the field of view of the first lens module, and the field of view of the third lens module may be smaller than the field of view of the second lens module, or larger than the field of view of the second lens module.

[0050] The embodiments of the present application do not limit the specific forms of the first lens module, the second lens module, and the third lens module. In some embodiments, the first lens module may be a telephoto lens module, the second lens module may be one of an ultra-wide-angle lens module and a wide-angle lens module, and the third lens module may be the other of the ultra-wide-angle lens module and the wide-angle lens module. In other embodiments, the first lens module may be a wide-angle lens module, and the second lens module may be an ultra-wide-angle lens module.

[0051] like Figure 2 As shown, the image processing method of the embodiment of the present application may include:

[0052] Step 201 : When the electronic device displays an image captured by the first lens module, obtain first color statistical information of a target lens module of the electronic device at a first moment, wherein the target lens module includes the second lens module.

[0053] In the embodiment of the present application, when the electronic device displays the image captured by the first lens module, the electronic device can be regarded as being in the display stage of the first lens module.

[0054] During the first lens module's display phase, the electronic device's target lens module can be in operation to perform color compensation on the first lens module, improving the first lens module's white balance processing effect. Meanwhile, lens modules other than the first and target lens modules can be in a deactivated state during the first lens module's display phase, thereby reducing the electronic device's power consumption.

[0055] In specific implementation, the target lens module may include part or all of the lens modules in the electronic device whose FOV is smaller than that of the first lens module. Therefore, the target lens module can be called a lens module with a large FOV, and the first lens module can be called a lens module with a small FOV.

[0056] In some embodiments, the operating state of the target lens module may be an always-on state. In this state, the operating frequency of the target lens module may be a default operating frequency of the target lens module, and the resolution of the target lens module may be a default resolution.

[0057] In other embodiments, the operating state of the target lens module can be at least one of a low power (LowFps) state and a low resolution (LowRes) state. In the low power state, the operating frequency of the target lens module can be 1 / k of the default operating frequency of the target lens module, where k is a power of 2 and is greater than 1, such as 4, 16, 256, etc. In the low resolution state, the resolution of the target lens module can be 1 / x of the default resolution of the target lens module, where x is an integer greater than 1, such as 2, 3, 4, 5, etc. It can be seen that compared to the normally open state, the electronic device consumes less power in the low power state and the low resolution state.

[0058] The number of target lens modules may be greater than or equal to 1. When the number of target lens modules is greater than 1, the working states of the target lens modules may be the same or different, which may be determined according to actual needs and is not limited in this embodiment of the present application.

[0059] In an embodiment of the present application, since the target lens module with a large FOV can obtain more color statistical information, the color statistical information of the target lens module can be used to perform color compensation on the first lens module with a small FOV. In this way, the white balance processing effect of the lens module with a small FOV can be improved.

[0060] For ease of understanding, the present embodiment of the present application uses the color compensation of the first lens module at a first moment as an example. The first moment is any image capture moment of the first lens module during the display phase of the first lens module. In other words, the image captured by the first lens module at any moment can be white-balanced using the image processing method of the present embodiment of the present application.

[0061] In a specific implementation, first color statistical information of a target lens module of an electronic device at a first moment may be obtained.

[0062] In the embodiment of the present application, the first color statistical information of a target lens module at a first moment can be understood as the color statistical information of a target image captured by the target lens module, where the target image can be a third image captured by the target lens module at the first moment, or a second image captured by the target lens module at its last historical image capture moment, or a copy of the second image. The last historical image capture moment of the target lens module, i.e., the moment of image capture of the target lens module that is before the first moment and closest to the first moment, can be referred to as the second moment.

[0063] In some embodiments, the target image captured by the target lens module can be predetermined.

[0064] In other embodiments, the specific representation of the target image captured by the target lens module may be determined based on a determination result of whether the first moment is the image capture moment of the target lens module.

[0065] It should be noted that, in actual applications, the operating frequency of each target lens module may be the same as or different from the operating frequency of the first lens module.

[0066] For a target lens module having the same operating frequency as the first lens module, its image acquisition moment is the same as the image acquisition moment of the first lens module. In this case, the first moment is also the image acquisition moment of the target lens module, that is, the target lens module will perform image acquisition at the first moment.

[0067] For a target lens module having a different operating frequency from that of the first lens module, its image acquisition moment is different from that of the first lens module. In this case, the first moment may or may not be the image acquisition moment of the target lens module.

[0068] If the first moment is the image acquisition moment of a target lens module, that is, the image acquisition moment of the target lens module includes the first moment, then the first color statistical information of the target lens module at the first moment can be understood as: the color statistical information of the third image acquired by the target lens module at the first moment.

[0069] If the first moment is the image capture moment of a target lens module, that is, the image capture moment of the target lens module includes the first moment, then the first color statistical information of the target lens module at the first moment can be understood as: the color statistical information of the second image captured by the target lens module at the second moment, or the color statistical information of a copy of the third image of the target lens module.

[0070] The embodiments of the present application do not limit the specific form of the color statistical information. In some embodiments, the color statistical information may include Gr information and Gb information. The method for obtaining color statistical information can be found in related art and will not be described here.

[0071] Step 202: Determine first color compensation information mapped by the first color statistical information based on a first color mapping relationship, wherein the first color mapping relationship is a color mapping relationship of the electronic device, and the color mapping relationship is a mapping relationship between the color statistical information of the target lens module and the color compensation information of the first lens module; the first color compensation information is the color compensation information of the first lens module of the electronic device at the first moment.

[0072] In an embodiment of the present application, the electronic device stores its own color mapping relationship, referred to as a first color mapping relationship, which can be used to describe the mapping relationship between color statistics of the target lens module of the electronic device and color compensation information of the first lens module of the electronic device.

[0073] The embodiment of the present application does not limit the representation method of the color mapping relationship. The color mapping relationship can be, but is not limited to, represented by a function, a network model, a mapping table, etc.

[0074] When the color mapping relationship is represented by a function, the independent variable of the function is the color statistics of the target lens module, and the dependent variable is the color compensation information of the first lens module. In this case, by substituting the first color statistics into the function, the color compensation information of the first lens module of the electronic device at the first moment can be obtained. This function can be called a prediction function.

[0075] When the color mapping relationship is represented by a network model, the input of the network model is the color statistics of the target lens module, and the output of the network model is the color compensation information of the first lens module. In this case, by inputting the first color statistics into the network model, the color compensation information of the first lens module of the electronic device at the first moment can be obtained.

[0076] In the case where the color mapping relationship is represented by a mapping table, the first color statistical information can be used to look up color compensation information of the first lens module of the electronic device at the first moment in the mapping table.

[0077] The embodiments of the present application do not limit the method for obtaining the first color mapping relationship. In some embodiments, the first color mapping relationship can be determined based on a reference color mapping relationship. For details, please refer to the relevant description below and will not be described here. In other embodiments, the first color mapping relationship can be generated based on test data of the electronic device in a manner similar to that of generating the reference color mapping relationship, which will not be described here. In other embodiments, the first color mapping relationship can be pre-written into the electronic device.

[0078] Step 203: Perform white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information.

[0079] In some embodiments, the first color compensation information can be directly used to determine the AWB gain of the first lens module at the first moment, such as taking the inverse of the first color compensation information to obtain the AWB gain, and then using the AWB gain of the first lens module at the first moment to perform white balance processing on the first image.

[0080] In other embodiments, the first image can be white balanced in combination with the first color compensation information and the color statistical information of the first lens module of the electronic device at the first moment. In a specific implementation, the first color compensation information and the color statistical information of the first lens module of the electronic device at the first moment can be weighted averaged to calculate the target color statistical information of the first lens module at the first moment, and then the AWB gain of the first lens module at the first moment can be determined using the target color statistical information, such as taking the inverse of the target color statistical information to obtain the AWB gain; and the first image can be white balanced using the AWB gain. In this embodiment, the AWB gain of the first lens module at the first moment is determined based on the color compensation information and its own color statistical information. In this way, compared with directly using the color compensation information to determine the AWB gain, the white balance processing effect can be further improved.

[0081] The image processing method of an embodiment of the present application can obtain first color statistical information of a target lens module with a large FOV at a first moment when an electronic device displays an image captured by a first lens module with a small FOV. Then, based on the mapping relationship between the color statistical information of the target lens module of the electronic device and the color compensation information of the first lens module, first color compensation information corresponding to the first color statistical information is determined, that is, the color compensation information of the first lens module at the first moment is obtained. Thereafter, white balance processing is performed on the first image captured by the first camera module of the electronic device at the first moment based on the first color compensation information. The embodiment of the present application can utilize the color statistical information of the target lens module with a large FOV to accurately determine the color compensation information of the first lens module with a small FOV through the mapping relationship between the color statistical information of the target lens module of the electronic device and the color compensation information of the first lens module, thereby accurately color compensating the lens module with a small FOV using the color statistical information of the target lens module with a large FOV. Since the target lens module with a large FOV can obtain more color statistical information, the white balance processing effect of the lens module with a small FOV can be improved.

[0082] The following describes in detail how to obtain the first color mapping relationship of the electronic device.

[0083] In some embodiments, before determining the first color compensation information mapped by the first color statistical information according to the first color mapping relationship, the method further includes:

[0084] Determining a color calibration coefficient of the electronic device relative to the reference device using the color statistical information of the electronic device and the color statistical information of the reference device;

[0085] The first color mapping relationship is determined by using the color calibration coefficient and a reference color mapping relationship, wherein the reference color mapping relationship is a color mapping relationship of the reference device.

[0086] In this embodiment, the first color mapping relationship is determined based on a reference color mapping relationship, which is a color mapping relationship of a reference device and is used to describe a mapping relationship between color statistics of a target lens module of the reference device and color compensation information of a first lens module of the reference device.

[0087] The electronic device and the reference device include the same lens module, and the lens modules included in both are lens modules from the same batch. The OTP parameters of the lens modules from the same batch are normally distributed. The OTP parameters of the lens module refer to the color-related characteristic parameters of the lens module, such as the gray point response results at various color temperatures and color statistical information. In the embodiments of the present application, the lens module whose OTP parameters conform to the mean of the normal distribution is referred to as the golden lens module, indicating that its color characteristics best represent the statistical laws of the entire batch of lens modules. The electronic device including the golden lens module is referred to as the reference device.

[0088] In a specific implementation, the color statistical information of the electronic device and the color statistical information of the reference device can be used to determine a color calibration coefficient of the electronic device relative to the reference device. The color calibration coefficient is then multiplied by the reference color mapping relationship to obtain a first color mapping relationship.

[0089] It should be noted that the color statistical information of the device used to determine the calibration coefficient is the color statistical information obtained before the device leaves the factory.

[0090] In some embodiments, the color mapping relationship may include a mapping relationship between the color statistical information of the target lens module under each color temperature classification and the color compensation information of the first lens module. In this embodiment, the color calibration coefficient of the electronic device relative to the reference device may include the color calibration coefficient of the electronic device relative to the reference device under each color temperature classification. The color statistical information of the electronic device includes the color statistical information of the electronic device under each color temperature classification, and the color statistical information of the reference device includes the color statistical information of the reference device under each color temperature classification.

[0091] The embodiments of the present application do not limit the method of color temperature classification. In some embodiments, color temperature classification may include, but is not limited to, high color temperature, interpolated color temperature, and low color temperature. Furthermore, high color temperature may include D75, D65, and D50; interpolated color temperature may include TL84 and CWF; and low color temperature may include H, A, and U30. In this embodiment, the entire natural color temperature range can be sampled according to eight groups of spectra, namely, H, A, U30, TL84, CWF, D50, D65, and D75, and the distribution pattern of the lens module's response to the gray area under different light sources can be statistically analyzed to obtain color statistical information of the lens module under each color temperature classification.

[0092] When determining the color calibration coefficient of an electronic device relative to a reference device, in some embodiments, the color statistical information of the same lens module of the electronic device and the reference device can be used to calculate the color calibration coefficient corresponding to the lens module, and use it as the color calibration coefficient of the electronic device relative to the reference device; in other embodiments, the color calibration coefficient corresponding to each lens module can be calculated first, and then the color calibration coefficient corresponding to each lens module can be used to determine the color calibration coefficient of the electronic device relative to the reference device.

[0093] The color calibration coefficient corresponding to the lens module can be obtained by dividing the color statistical information of the lens module of the electronic device by the color statistical information of the lens module of the reference device.

[0094] In this embodiment, the first color mapping relationship is determined based on a reference color mapping relationship. Because the color characteristics of the lens module of the reference device can represent the statistical laws of the entire batch of lens modules, the color mapping relationship of the electronic device is determined based on the color mapping relationship of the reference device. In this way, the color mapping accuracy of the color mapping relationship of the electronic device can be improved.

[0095] The following is a detailed description of how to obtain the reference color mapping relationship.

[0096] In some embodiments, before determining the first color mapping relationship by using the color calibration coefficient and the reference color mapping relationship, the method may further include:

[0097] Constructing a first matrix using color statistical information of the target lens module of the reference device;

[0098] Constructing a second matrix using the color statistical information of the first lens module of the reference device, wherein the dimension of the first matrix is ​​greater than or equal to the dimension of the second matrix;

[0099] Constructing the initial color mapping relationship;

[0100] The initial color mapping relationship is trained based on the first matrix and the second matrix to obtain the reference color mapping relationship.

[0101] In this embodiment, the input and output formats of the initial color mapping relationship are both matrices, but this does not limit the input and output formats of the initial color mapping relationship. In other embodiments, the input and output formats of the initial color mapping relationship can be vectors, etc.

[0102] In a specific implementation, the first matrix can be constructed using the color statistical information of the target lens modules of the reference device. When the number of target lens modules is 1, the first matrix includes all the color statistical information of the target lens modules of the reference device. When the number of target lens modules is greater than 1, the first matrix may include all the color statistical information of at least one target lens module of the reference device, in addition to including the color statistical information of all target lens modules. In other words, when the number of target lens modules is greater than 1, the first matrix may only include partial color statistical information for some target lens modules.

[0103] For easier understanding, the following examples are provided:

[0104] Assume that the color statistical information includes Gr and Gb, and the target lens module includes an ultra-wide-angle lens module (hereinafter referred to as UW) and a wide-angle lens module (hereinafter referred to as W). Then, the first matrix may include Gr+Gb of UW and Gr+Gb of W; or Gr+Gb of UW and Gr of W; or Gr+Gb of UW and Gb of W; or Gr of UW and Gr+Gb of W; or Gb of UW and Gr+Gb of W; or Gb of UW and Gr+Gb of W.

[0105] The second matrix can be constructed using the color statistical information of the first lens module of the reference device. The first matrix includes all the color statistical information of the first lens module of the reference device.

[0106] It can be understood that when the number of the target lens module is 1, the dimensions of the first matrix and the second matrix are equal.

[0107] When the number of target lens modules is greater than 1, the dimension of the first matrix is ​​greater than that of the second matrix. In this case, the trained reference color mapping relationship can map low-dimensional color compensation information using high-dimensional color statistics, thereby reducing linear inseparability and improving color mapping accuracy.

[0108] In addition, an initial color mapping relationship can also be constructed. Then, the first matrix is ​​used as the input of the initial color mapping relationship, and the second matrix is ​​used as the calibration output of the initial color mapping relationship. The initial color mapping relationship is trained until the training stop condition is met to obtain a reference color mapping relationship. The embodiments of the present application do not limit the specific form of the training stop condition. In some embodiments, the training stop condition can be expressed as: the norm of the matrix output by the initial color mapping relationship and the second matrix is ​​the smallest, that is, the matrix output by the initial color mapping relationship is close to the second matrix.

[0109] In this embodiment, an initial color mapping relationship can be constructed in advance, and then a second matrix is ​​constructed based on the color statistical information of the target lens module of the reference device and the color statistical information of the first lens module of the reference device. The initial mapping relationship is trained to obtain a reference color mapping relationship. In this way, the reliability of the reference color mapping relationship can be improved.

[0110] In other embodiments, the reference color mapping relationship can be obtained by performing regression fitting on the color statistical information of the target lens module of the reference device and the color statistical information of the first lens module of the reference device. In this way, the acquisition of the reference color mapping relationship can be simplified.

[0111] The first color statistical information of the target lens module of the electronic device at the first moment is described in detail below.

[0112] In some embodiments, before obtaining first color statistical information of the target lens module of the electronic device at a first moment, the method further includes:

[0113] When the electronic device displays an image captured by the first lens module, adjusting the target lens module to a second operating frequency, wherein the second operating frequency is less than the first operating frequency, and the first operating frequency is the operating frequency of the first lens module;

[0114] Determining an image acquisition time of the first lens module and an image acquisition time of the target lens module respectively by using the first operating frequency and the second operating frequency;

[0115] The obtaining of first color statistical information of the target lens module of the electronic device at a first moment includes at least one of the following:

[0116] When the image acquisition moment of the target lens module does not include the first moment, color statistical information of a second image acquired by the target lens module at a second moment is determined as the first color statistical information, wherein the second moment is a moment before the first moment and closest to the first moment among the image acquisition moments of the target lens module;

[0117] In a case where the image acquisition moment of the target lens module includes the first moment, color statistical information of the third image acquired by the target lens module at the first moment is determined as the first color statistical information.

[0118] In this embodiment, during the display stage of the first lens module, the target lens module is mainly used to perform color compensation on the first lens module. The target lens module can be controlled to be in a low-frequency working state to reduce the hardware power consumption of the electronic device.

[0119] In a specific implementation, the operating frequency of the target lens module can be adjusted based on the operating frequency of the first lens module, and the operating frequency of the target lens module can be adjusted to be lower than the operating frequency of the first lens module. It should be noted that when the number of target lens modules is greater than one, the operating frequencies of different target lens modules can be the same or different, and can be set according to actual circumstances, and this embodiment of the application does not limit this.

[0120] Once the operating frequency of the lens module is determined, its image acquisition time is also determined accordingly. It will be appreciated that if the operating frequency of the target lens module is lower than that of the first lens module, the image acquisition time of the target lens module will be less than that of the first lens module. Therefore, the first time may or may not be the image acquisition time of the target lens module.

[0121] In the case where the first moment is the image acquisition moment of the target lens module, the first color statistical information of the target lens module at the first moment is the color statistical information of the third image acquired by the target lens module at the first moment. That is to say, in this case, color compensation is performed on the first lens module at the first moment by using the color statistical information of the target lens module at the first moment. Since the acquisition moment of the third image is the same as that of the first image, the accuracy of the color compensation of the first lens module can be improved.

[0122] In the case that the first moment is not the image acquisition moment of the target lens module, the first color statistical information of the target lens module at the first moment is the color statistical information of the third image acquired by the target lens module at the second moment. That is to say, in this case, color compensation is performed on the first lens module at the first moment through the color statistical information of the historical image acquisition moment of the target lens module closest to the first moment. Since the acquisition moment of the last image acquired by the target lens module is close to the first moment, the accuracy of the color compensation of the first lens module can be improved.

[0123] Through the above method, when color compensation is performed on the first lens module at the first moment, regardless of whether the first moment is the image acquisition moment of the target lens module, the image acquired by the target lens module can be used to perform color compensation on the first lens module. In this way, the reliability of the color compensation of the first lens module can be improved. In addition, if the first moment is the image acquisition moment of the target lens module, the color compensation of the first lens module is performed using the color statistical information of the third image acquired by the target lens module at the first moment. Since the acquisition moment of the third image is the same as the acquisition moment of the first image, the accuracy of the color compensation of the first lens module can be improved. If the first moment is not the image acquisition moment of the target lens module, the color compensation of the first lens module is performed using the color statistical information of the last image acquired by the target lens module. Since the acquisition moment of the last image acquired by the target lens module is close to the first moment, the accuracy of the color compensation of the first lens module can be improved.

[0124] In some embodiments, obtaining first color statistical information of a target lens module of the electronic device at a first moment includes:

[0125] When the image acquisition moment of the target lens module includes the first moment, obtaining first color statistical information of the target lens module of the electronic device at the first moment;

[0126] After respectively determining the image acquisition time of the first lens module and the image acquisition time of the target lens module by using the first operating frequency and the second operating frequency, the method further includes:

[0127] When the image acquisition moment of the target lens module does not include the first moment, performing white balance processing on the first image acquired by the first lens module of the electronic device at the first moment using the second color compensation information;

[0128] The second color compensation information is color compensation information of the first lens module at a third moment, and the third moment is a moment before the first moment in the image acquisition moments of the first lens module.

[0129] It should be noted that, when the number of target lens modules is greater than 1, as long as there is a target lens module whose image acquisition moment includes the first moment, the image acquisition moment of the target lens module of the visual electronic device includes the first moment; only when the image acquisition moments of all target lens modules do not include the first moment, the image acquisition moment of the target lens module of the visual electronic device does not include the first moment.

[0130] In this embodiment, when the image acquisition moment of the target lens module of the electronic device includes the first moment, white balance processing can be performed on the first image through steps 101 to 103.

[0131] If the image capture moment of the target lens module of the electronic device does not include the first moment, if white balancing is performed on the first image through steps 101 to 103, since the first color statistical information of the target lens module of the electronic device at the first moment is represented by the color statistical information of the second image captured by the target lens module of the electronic device at the second moment, the first color compensation information calculated is the same as the second color compensation information calculated using the second color statistical information of the target lens module at the second moment. Based on this, in this case, the second color compensation information can be directly reused to perform white balancing on the first image. This reduces the number of color compensation information calculations, thereby further reducing the operating power consumption of the electronic device.

[0132] For easier understanding, the following examples are provided:

[0133] Assume that the first lens module is a telephoto lens module, the target lens module includes an ultra-wide-angle lens module and a wide-angle lens module, the operating frequency of the wide-angle lens module is 1 / 2 of the telephoto lens module, and the operating frequency of the ultra-wide-angle lens module is 1 / 3 of the telephoto lens module.

[0134] The image selection strategy for color compensation of the first lens module is as follows: Figure 3 As shown. Figure 3 In , the blank box represents the image captured by the lens module, and the filled box represents the copy of the image captured last time. Based on the corresponding relationship between the working frequencies of the three camera modules, such as Figure 3 As shown, for every 2 images captured by the telephoto lens module, the wide-angle lens module captures 1 image; for every 3 images captured by the telephoto lens module, the ultra-wide-angle lens module captures 1 image.

[0135] For the first frame image captured by the telephoto lens module, when performing white balance compensation thereon, the color compensation information of the first frame image captured by the wide-angle lens module and the first frame image captured by the ultra-wide-angle lens module can be calculated using the color statistical information of the first frame image captured by the telephoto lens module.

[0136] For the second frame of image captured by the telephoto lens module, when performing white balance compensation on it, since the image capture time is not the image capture time of the wide-angle lens module and the ultra-wide-angle lens module, the color statistical information of the first frame of image captured by the wide-angle lens module and the first frame of image captured by the ultra-wide-angle lens module can be used to calculate the color compensation information of the second frame of image captured by the telephoto lens module.

[0137] It can be seen that the color compensation information of the first frame image and the second frame image captured by the telephoto lens module are the same. Therefore, the color compensation information of the first frame image captured by the telephoto lens module can be directly reused to perform white balance processing on the second frame image captured by the telephoto lens module.

[0138] Through this embodiment, when the image acquisition moment of the target lens module of the electronic device includes the first moment, color compensation information of the first image can be obtained through calculation; when the image acquisition moment of the target lens module of the electronic device does not include the first moment, the color compensation information obtained by the last calculation can be directly reused as the color compensation information of the first image. In this way, the number of calculations of the color compensation information can be reduced, thereby further reducing the operating power consumption of the electronic device.

[0139] In some embodiments of the present application, the method may further include:

[0140] When the electronic device displays the image captured by the first lens module, adjusting the resolution of the second lens module from the first resolution to the second resolution;

[0141] The second resolution is smaller than the first resolution, and the first resolution is the resolution of the second lens module when the electronic device displays an image captured by the second lens module.

[0142] In this embodiment, considering that the resolution required for white balance processing is not high, during the display stage of the first lens module, the target lens module can be controlled to be in a low-resolution working state, thereby reducing the hardware power consumption of the electronic device.

[0143] The following is a detailed description of the determination of the target lens module in the embodiment of the present application.

[0144] In some embodiments, the target lens module further includes a third lens module of the electronic device, wherein the field of view of the third lens module is larger than the field of view of the first lens module.

[0145] In this embodiment, some or all lens modules of the electronic device whose FOV is larger than the FOV of the first lens module can be directly determined as target lens modules. In this way, the method of determining the target lens module can be simplified and the efficiency of determining the target lens module can be improved.

[0146] In some embodiments, the electronic device further comprises a third lens module, wherein the field of view of the third lens module is larger than the field of view of the first lens module;

[0147] Before obtaining the first color statistical information of the target lens module of the electronic device at the first moment, the method may further include:

[0148] Obtaining an angle between a target component of a first hyperplane and a target component of a second hyperplane, wherein the target component includes at least one of a principal component and a secondary component; the first hyperplane is a hyperplane formed by response curves of each color channel corresponding to the second lens module; and the second hyperplane is a hyperplane formed by response curves of each color channel corresponding to the third lens module;

[0149] When the included angle is less than the included angle threshold, determining the second lens module as the target lens module;

[0150] When the included angle is greater than or equal to the included angle threshold, the second lens module and the third lens module are determined as the target lens module.

[0151] In this embodiment, the electronic device includes at least two lens modules with FOVs smaller than the first lens module, and the target lens module can be selected and determined from these at least two lens modules.

[0152] In a specific implementation, for each of the at least two lens modules, a hyperplane formed by the corresponding color channel response curves can be utilized. Subsequently, for any two of the at least two lens modules, at least one of the following items can be calculated: the angle of the principal component of the corresponding surface, and the angle of the secondary component of the corresponding surface. These angles are then compared with their corresponding angle thresholds, and the target lens module is determined based on the comparison results.

[0153] Specifically, if the included angle is less than the corresponding included angle threshold, it means that the difference in the response functions of the two lens modules is small, and one lens module can be selected from the two lens modules as the target lens module. If the included angle is greater than the corresponding included angle threshold, it means that the difference in the response functions of the two lens modules is large, and both lens modules can be determined as the target lens modules.

[0154] The angle thresholds corresponding to the main component and the secondary component can be pre-set. In one example, the angle threshold corresponding to the main component can be 2°, and the angle threshold corresponding to the secondary component can be 5°, but the present invention is not limited thereto. The color channels can be red (R), green (G), and blue (B) channels.

[0155] By determining the target lens module in the above manner, the data volume of the first color statistical information can be reduced, thereby reducing the color mapping burden.

[0156] In some embodiments, when the target lens module is the second lens module, before performing white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information, the method further includes:

[0157] When the color temperature set includes the ambient color temperature of the electronic device, determining a target compensation weight corresponding to the second lens module using a reference compensation weight corresponding to the second lens module, wherein the target compensation weight is less than the reference compensation weight; the color temperature set includes at least one color temperature, and for each color temperature of the at least one color temperature, statistical information of the second lens module and the first lens module exhibits metamerism at the color temperature;

[0158] The performing white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information includes:

[0159] multiplying the first color compensation information and the target compensation weight to obtain target color compensation information;

[0160] Based on the target color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

[0161] In this embodiment, the number of the target lens module is 1, and the probability of the statistical information of the target lens module and the first lens module having metamerism is high.

[0162] It may be first determined whether the statistical information of the target lens module and the first lens module has a metamerism problem under the current ambient color temperature of the electronic device.

[0163] In a specific implementation, a color temperature set may be obtained in advance. The color temperature set includes all color temperatures corresponding to the metamerism problem in the statistical information of the target lens module and the first lens module. The color temperature set may be determined by, but is not limited to, the following methods:

[0164] Obtaining a mapping relationship between sampling points corresponding to the color statistical information of the first lens module and sampling points corresponding to the color statistical information of the target lens module;

[0165] When there are at least two sampling points in the first lens module that are mapped to the first sampling point corresponding to the target lens module, it is determined that the color statistical information of the first lens module and the color statistical information of the target lens module have metamerism at the color temperature corresponding to the first sampling point.

[0166] After obtaining the color temperature set, it is possible to determine whether the statistical information of the target lens module and the first lens module has metamerism problems under the current color temperature of the electronic device by determining whether the current ambient color temperature of the electronic device is within the color temperature set.

[0167] If the color temperature set includes the current ambient color temperature of the electronic device, it can be determined that the statistical information of the target lens module and the first lens module has metamerism under the current ambient color temperature. When using the first color compensation information to perform white balance processing on the first image, the compensation weight corresponding to the target lens module can be reduced to reduce the contribution of the first color compensation information to the white balance processing of the first image, thereby reducing the impact of the metamerism problem on the white balance processing and improving the white balance processing effect.

[0168] If the color temperature set does not include the current ambient color temperature of the electronic device, it can be determined that the statistical information of the target lens module and the first lens module does not have a metamerism problem under the current ambient color temperature. The compensation weight corresponding to the target lens module can be left unchanged, and the contribution of the first color compensation information to the white balance processing of the first image can be used.

[0169] In the above manner, when the statistical information of the target lens module and the first lens module has metamerism under the current ambient color temperature, the contribution of the first color compensation information to the white balance processing of the first image can be reduced by reducing the compensation weight corresponding to the target lens module, thereby reducing the impact of the metamerism problem on the white balance processing and improving the white balance processing effect.

[0170] It should be noted that the various optional embodiments introduced in the embodiments of the present application can be implemented in combination with each other or separately if they do not conflict with each other, and the embodiments of the present application do not limit this.

[0171] To facilitate understanding of the image processing method provided by the above embodiment, the above image processing method is described below using a specific scenario embodiment.

[0172] In the following scenario embodiment, the first lens module is a telephoto lens module, and the target lens module includes at least one of an ultra-wide-angle lens module and a wide-angle lens module. The color statistical information is simply referred to as statistical information.

[0173] The image processing method of this embodiment may include the following steps:

[0174] Step 1: Because the calculation of color mapping does not require a very high frequency, and the resolution of the statistical information required for AWB calculation does not need to be very high, in some embodiments, such as Figure 4aAs shown, the always-on stage can be replaced with a low frame rate (LowFps) and low resolution (LowRes) stage. For example, the frame rate can be compressed to 1 / x of the original (x = 2, 3, 4, 5) and the resolution can be compressed to 1 / k of the original (k = 4, 16, 256).

[0175] Step 2: Based on step 1, a special processing strategy is applied to the telephoto lens module. Figure 4b .

[0176] Because the telephoto lens module has a limited FOV, there's a chance that insufficient grayscale in the telephoto lens' statistical information will result in AWB color casts. Using only the wide-angle lens module for color mapping also creates the possibility of many-to-one metamerism. This problem stems from the fact that predicting data in two-dimensional space is more prone to linear inseparability. Step two innovatively addresses classification and regression issues by constructing higher-dimensional data.

[0177] Because the ultra-wide-angle and wide-angle lens modules have different response functions, the two-dimensional subspaces spanned by the statistical information of their respective integrations with the light source spectrum are also linearly independent. Step 2 uses the statistical information of the wide-angle and ultra-wide-angle lens modules to construct a four-dimensional space. The Gr and Gb information of the original wide-angle lens module is upgraded to wide-angle Gr, Gb and ultra-wide-angle Gr, Gb information to solve the problem of linearly inseparable metamerism in the two-dimensional subspace.

[0178] The main strategy of step two is to call the ultra-wide-angle and wide-angle lens modules to be in the low frame rate and low resolution mode of MCSS at the same time when the telephoto lens module is sending the display. The frame rate of the ultra-wide-angle lens module can be lower than that of the wide-angle lens module.

[0179] Assume that the operating frequency of the wide-angle lens module is 1 / 2 of the telephoto lens module, and the operating frequency of the ultra-wide-angle lens module is 1 / 3 of the telephoto lens module. The image selection strategy of the three lens modules is as follows: Figure 3 shown.

[0180] Step 3. The integral of different reflection spectra and lens module response functions has a priori distribution law in the entire gray area space. Take a picture of a gray card (the reflectivity is equal everywhere in the range of 380nm-800nm) under the light source spectrum of 2000K-10000K. You can sample the entire natural color temperature range according to eight groups of spectra, including H, A, U30, TL84, CWF, D50, D65, and D75, and count the distribution law of each lens module's response to the gray area under different light sources to obtain the statistical information of each lens module at different color temperatures. For gray area selection, please refer to Figure 4c .

[0181] Step 4 and step 3 need to be calibrated using a Golden phone, which can guarantee the color effect of this batch of lens modules to the greatest extent. That is, it is necessary to pre-process the data of the Golden module (the lens module of the Golden phone) and the OTP module (the lens module of the electronic device) to obtain the calibration coefficient ratio of the two; since OTP and Golden are currently calibrated according to high color temperature, low color temperature, and interpolated color temperature, it is necessary to calculate the calibration coefficient ratio according to different color temperatures. D75, D65, and D50 can be classified as high color temperature, TL84, and CWF can be classified as interpolated color temperature, and H, A, and U30 can be classified as low color temperature. For specific ideas, please refer to Figure 5 .

[0182] Step 5. According to steps 2 to 4, construct the four-dimensional input matrix inputMatrix and the two-dimensional output matrix outputMatrix. The specific form is as follows:

[0183] Input matrix:

[0184] Output matrix:

[0185] The parameters in the above matrix are all the statistical information values ​​of Golden mobile phone. uw It represents the Gr value of the Golden phone's ultra-wide-angle lens module; Gr w It represents the Gr value of the wide-angle lens module of the Golden mobile phone; Gr tele It represents the Gr value of the telephoto lens module of the Golden mobile phone.

[0186] Step 6: Construct a prediction function (i.e., color mapping relationship) and perform regression fitting on the input and output matrices of step 5. Ensure that the result of the input matrix passing through the prediction function is the smallest L2 norm with the output matrix. At this point, the prediction matrix PreFunc is obtained. The process can be expressed as follows, and the flowchart can be found in Figure 6 :

[0187] argmin||PreFunc(inputMatrix)-outputMatrix|| 2

[0188] Step 7: The process of deploying the mapping is to write the prediction function PreFunc into the program, and multiply the ratio of the corresponding effective mobile phone (electronic device) by the prediction function to obtain the prediction function of the mobile phone itself.

[0189] Step 8: The mapping process is to feed the statistical information of the ultra-wide-angle and wide-angle lens modules into step 7 to obtain the color prediction value of the telephoto lens module, and then use the color prediction value to determine AWBGain. For specific implementation, please refer to Figure 7 .

[0190] Step 9. If the measured differences in the response functions of different lens modules satisfy a certain statistical range, that is, the angles between the principal components of the hyperplanes of different lens groups are less than the corresponding angle threshold, or the angles between the secondary principal components of the hyperplanes of different lens groups are less than the corresponding angle threshold, the input matrix of step 5 can be simplified. For example, the statistical information of the ultra-wide-angle or wide-angle lens module can be used alone to perform color mapping on the telephoto lens module. The rest of the steps still need to meet the requirements of steps 3 to 8.

[0191] Figure 8a and Figure 8b It shows the distribution of gray areas before and after color mapping in two-dimensional space. Figure 8a This is the distribution of the gray area of ​​the ultra-wide-angle or wide-angle lens module. Figure 8b The figure shows the distribution of the grayscale area of ​​the telephoto lens module. The horizontal and vertical axes are Gr and Gb, respectively, and the blue dots represent the sampling coordinates of the grayscale area at different color temperatures. The red auxiliary grid is distorted or stretched near the calibration points, indicating that the prediction function is highly localized; the mapping at one color temperature does not affect the sampling results at other color temperatures. Furthermore, based on the distribution of the mapped sampling points, color temperatures prone to metamerism are identified and treated with special measures, specifically reducing the color compensation weights for the ultra-wide-angle or wide-angle lens modules.

[0192] This scenario embodiment improves the existing MCSS hard synchronization system and proposes a new color mapping solution. This invention uses the collaborative work of multiple lens modules to obtain higher-dimensional statistical information and perform color correction on the lens module with a small FOV of the target lens module. While significantly reducing power consumption, it also solves the color jump problem caused by lens module switching during the SAT process and improves the color accuracy of lens modules with a small FOV.

[0193] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0194] like Figure 9 As shown, the image processing device in the embodiment of the present application may include:

[0195] A first acquisition module 901 is configured to acquire, when the electronic device displays an image captured by the first lens module, first color statistical information of a target lens module of the electronic device at a first moment, wherein the target lens module includes the second lens module;

[0196] A first determining module 902 is configured to determine first color compensation information mapped to the first color statistical information based on a first color mapping relationship, wherein the first color mapping relationship is a color mapping relationship of the electronic device, the color mapping relationship being a mapping relationship between the color statistical information of the target lens module and the color compensation information of the first lens module; and the first color compensation information being color compensation information of the first lens module of the electronic device at the first moment;

[0197] The first white balance processing module 903 is configured to perform white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information.

[0198] In some embodiments, the apparatus further comprises:

[0199] a second determining module, configured to determine a color calibration coefficient of the electronic device relative to the reference device by using the color statistical information of the electronic device and the color statistical information of the reference device;

[0200] The third determining module is configured to determine the first color mapping relationship by using the color calibration coefficient and a reference color mapping relationship, wherein the reference color mapping relationship is a color mapping relationship of the reference device.

[0201] In some embodiments, the apparatus further comprises:

[0202] A first construction module is configured to construct a first matrix using color statistical information of the target lens module of the reference device;

[0203] A second construction module is configured to construct a second matrix using the color statistical information of the first lens module of the reference device, wherein the dimension of the first matrix is ​​greater than or equal to the dimension of the second matrix;

[0204] The third building module is used to build an initial color mapping relationship;

[0205] A training module is used to train the initial color mapping relationship based on the first matrix and the second matrix to obtain the reference color mapping relationship.

[0206] In some embodiments, the apparatus further comprises:

[0207] a first adjustment module, configured to adjust the target lens module to a second operating frequency when the electronic device displays an image captured by the first lens module, wherein the second operating frequency is lower than the first operating frequency, and the first operating frequency is the operating frequency of the first lens module;

[0208] a fourth determining module, configured to determine an image acquisition moment of the first lens module and an image acquisition moment of the target lens module respectively by using the first operating frequency and the second operating frequency;

[0209] The first acquisition module is specifically configured to perform at least one of the following:

[0210] When the image acquisition moment of the target lens module does not include the first moment, color statistical information of a second image acquired by the target lens module at a second moment is determined as the first color statistical information, wherein the second moment is a moment before the first moment and closest to the first moment among the image acquisition moments of the target lens module;

[0211] In a case where the image acquisition moment of the target lens module includes the first moment, color statistical information of the third image acquired by the target lens module at the first moment is determined as the first color statistical information.

[0212] In some embodiments, the first acquisition module is specifically configured to:

[0213] When the image acquisition moment of the target lens module includes the first moment, obtaining first color statistical information of the target lens module of the electronic device at the first moment;

[0214] The device further comprises:

[0215] a second white balance processing module, configured to perform white balance processing on a first image captured by the first lens module of the electronic device at the first moment using second color compensation information when the image capture moment of the target lens module does not include the first moment;

[0216] The second color compensation information is color compensation information of the first lens module at a third moment, and the third moment is a moment before the first moment in the image acquisition moments of the first lens module.

[0217] In some embodiments, the apparatus further comprises:

[0218] a second adjustment module, configured to adjust the resolution of the second lens module from a first resolution to a second resolution when the electronic device displays the image captured by the first lens module;

[0219] The second resolution is smaller than the first resolution, and the first resolution is the resolution of the second lens module when the electronic device displays an image captured by the second lens module.

[0220] In some embodiments, the target lens module further includes a third lens module of the electronic device, wherein the field of view of the third lens module is larger than the field of view of the first lens module.

[0221] In some embodiments, the electronic device further comprises a third lens module, wherein the field of view of the third lens module is larger than the field of view of the first lens module;

[0222] The device further comprises:

[0223] a second acquisition module, configured to acquire an angle between a target component of the first hyperplane and a target component of the second hyperplane, wherein the target component includes at least one of a principal component and a secondary component; the first hyperplane is a hyperplane formed by response curves of the color channels corresponding to the second lens module; and the second hyperplane is a hyperplane formed by response curves of the color channels corresponding to the third lens module;

[0224] a fifth determining module, configured to determine the second lens module as the target lens module when the included angle is less than an included angle threshold;

[0225] The sixth determining module is configured to determine the second lens module and the third lens module as the target lens module when the included angle is greater than or equal to the included angle threshold.

[0226] In some embodiments, when the target lens module is the second lens module, the apparatus further includes:

[0227] a sixth determining module, configured to, when a color temperature set includes the ambient color temperature of the electronic device, determine a target compensation weight corresponding to the second lens module using a reference compensation weight corresponding to the second lens module, wherein the target compensation weight is less than the reference compensation weight; the color temperature set includes at least one color temperature, and for each color temperature of the at least one color temperature, statistical information of the second lens module and the first lens module exhibits metamerism at the color temperature;

[0228] The first white balance processing module includes:

[0229] an acquiring unit, configured to multiply the first color compensation information and the target compensation weight to obtain target color compensation information;

[0230] The white balance processing unit is configured to perform white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the target color compensation information.

[0231] The image processing device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0232] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0233] The image processing device provided in the embodiment of the present application can implement each process of the method embodiment. To avoid repetition, it will not be described here.

[0234] Alternatively, as Figure 10 As shown, an embodiment of the present application further provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores a program or instruction that can be run on the processor 1001, and when the program or instruction is executed by the processor 1001, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0235] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0236] Figure 11 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0237] The electronic device 1100 includes but is not limited to components such as a radio frequency unit 1101 , a network module 1102 , an audio output unit 1103 , an input unit 1104 , a sensor 1105 , a display unit 1106 , a user input unit 1107 , an interface unit 1108 , a memory 1109 , and a processor 1110 .

[0238] Those skilled in the art will understand that the electronic device 1100 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0239] The processor 1010 is configured to:

[0240] When the electronic device displays an image captured by the first lens module, obtaining first color statistical information of a target lens module of the electronic device at a first moment, wherein the target lens module includes the second lens module;

[0241] Determining first color compensation information mapped to the first color statistical information based on a first color mapping relationship, wherein the first color mapping relationship is a color mapping relationship of the electronic device, the color mapping relationship being a mapping relationship between the color statistical information of the target lens module and the color compensation information of the first lens module; and the first color compensation information being color compensation information of the first lens module of the electronic device at the first moment;

[0242] Based on the first color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

[0243] In some embodiments, the processor 1010 is configured to:

[0244] Determining a color calibration coefficient of the electronic device relative to the reference device using the color statistical information of the electronic device and the color statistical information of the reference device;

[0245] The first color mapping relationship is determined by using the color calibration coefficient and a reference color mapping relationship, wherein the reference color mapping relationship is a color mapping relationship of the reference device.

[0246] In some embodiments, the processor 1010 is configured to:

[0247] Constructing a first matrix using color statistical information of the target lens module of the reference device;

[0248] Constructing a second matrix using the color statistical information of the first lens module of the reference device, wherein the dimension of the first matrix is ​​greater than or equal to the dimension of the second matrix;

[0249] Constructing the initial color mapping relationship;

[0250] The initial color mapping relationship is trained based on the first matrix and the second matrix to obtain the reference color mapping relationship.

[0251] In some embodiments, the processor 1010 is configured to:

[0252] When the electronic device displays an image captured by the first lens module, adjusting the target lens module to a second operating frequency, wherein the second operating frequency is less than the first operating frequency, and the first operating frequency is the operating frequency of the first lens module;

[0253] Determining an image acquisition time of the first lens module and an image acquisition time of the target lens module respectively by using the first operating frequency and the second operating frequency;

[0254] When the image acquisition moment of the target lens module does not include the first moment, color statistical information of a second image acquired by the target lens module at a second moment is determined as the first color statistical information, wherein the second moment is a moment before the first moment and closest to the first moment among the image acquisition moments of the target lens module;

[0255] In a case where the image acquisition moment of the target lens module includes the first moment, color statistical information of the third image acquired by the target lens module at the first moment is determined as the first color statistical information.

[0256] In some embodiments, the processor 1010 is configured to:

[0257] When the image acquisition moment of the target lens module includes the first moment, obtaining first color statistical information of the target lens module of the electronic device at the first moment;

[0258] When the image acquisition moment of the target lens module does not include the first moment, performing white balance processing on the first image acquired by the first lens module of the electronic device at the first moment using the second color compensation information;

[0259] The second color compensation information is color compensation information of the first lens module at a third moment, and the third moment is a moment before the first moment in the image acquisition moments of the first lens module.

[0260] In some embodiments, the processor 1010 is configured to:

[0261] When the electronic device displays the image captured by the first lens module, adjusting the resolution of the second lens module from the first resolution to the second resolution;

[0262] The second resolution is smaller than the first resolution, and the first resolution is the resolution of the second lens module when the electronic device displays an image captured by the second lens module.

[0263] In some embodiments, the target lens module further includes a third lens module of the electronic device, wherein the field of view of the third lens module is larger than the field of view of the first lens module.

[0264] In some embodiments, the electronic device further comprises a third lens module, wherein the field of view of the third lens module is larger than the field of view of the first lens module;

[0265] The processor 1010 is configured to:

[0266] Obtaining an angle between a target component of a first hyperplane and a target component of a second hyperplane, wherein the target component includes at least one of a principal component and a secondary component; the first hyperplane is a hyperplane formed by response curves of each color channel corresponding to the second lens module; and the second hyperplane is a hyperplane formed by response curves of each color channel corresponding to the third lens module;

[0267] When the included angle is less than the included angle threshold, determining the second lens module as the target lens module;

[0268] When the included angle is greater than or equal to the included angle threshold, the second lens module and the third lens module are determined as the target lens module.

[0269] In some embodiments, when the target lens module is the second lens module, the processor 1010 is configured to:

[0270] When the color temperature set includes the ambient color temperature of the electronic device, determining a target compensation weight corresponding to the second lens module using a reference compensation weight corresponding to the second lens module, wherein the target compensation weight is less than the reference compensation weight; the color temperature set includes at least one color temperature, and for each color temperature of the at least one color temperature, statistical information of the second lens module and the first lens module exhibits metamerism at the color temperature;

[0271] multiplying the first color compensation information and the target compensation weight to obtain target color compensation information;

[0272] Based on the target color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

[0273] The electronic device 1100 provided in the embodiment of the present application can implement each process of the method embodiment, and to avoid repetition, they will not be described here.

[0274] It should be understood that in an embodiment of the present application, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0275] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0276] Processor 1110 may include one or more processing units. Optionally, processor 1110 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1110.

[0277] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0278] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0279] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0280] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0281] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0282] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0283] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0284] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a first lens module and a second lens module, wherein the field of view of the first lens module is smaller than the field of view of the second lens module; the method includes: When the electronic device displays an image captured by the first lens module, obtaining first color statistical information of a target lens module of the electronic device at a first moment, wherein the target lens module includes the second lens module; Determining first color compensation information mapped to the first color statistical information based on a first color mapping relationship, wherein the first color mapping relationship is a color mapping relationship of the electronic device, the color mapping relationship being a mapping relationship between the color statistical information of the target lens module and the color compensation information of the first lens module; and the first color compensation information being color compensation information of the first lens module of the electronic device at the first moment; Based on the first color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

2. The method according to claim 1, characterized in that Before determining the first color compensation information mapped by the first color statistical information according to the first color mapping relationship, the method further includes: Determining a color calibration coefficient of the electronic device relative to the reference device using the color statistical information of the electronic device and the color statistical information of the reference device; The first color mapping relationship is determined by using the color calibration coefficient and a reference color mapping relationship, wherein the reference color mapping relationship is a color mapping relationship of the reference device.

3. The method according to claim 2, characterized in that Before determining the first color mapping relationship by using the color calibration coefficient and the reference color mapping relationship, the method further includes: Constructing a first matrix using color statistical information of the target lens module of the reference device; Constructing a second matrix using the color statistical information of the first lens module of the reference device, wherein the dimension of the first matrix is ​​greater than or equal to the dimension of the second matrix; Constructing the initial color mapping relationship; The initial color mapping relationship is trained based on the first matrix and the second matrix to obtain the reference color mapping relationship.

4. The method according to claim 1, wherein Before obtaining the first color statistical information of the target lens module of the electronic device at the first moment, the method further includes: When the electronic device displays an image captured by the first lens module, adjusting the target lens module to a second operating frequency, wherein the second operating frequency is less than the first operating frequency, and the first operating frequency is the operating frequency of the first lens module; Determining an image acquisition time of the first lens module and an image acquisition time of the target lens module respectively by using the first operating frequency and the second operating frequency; The obtaining of first color statistical information of the target lens module of the electronic device at a first moment includes at least one of the following: When the image acquisition moment of the target lens module does not include the first moment, color statistical information of a second image acquired by the target lens module at a second moment is determined as the first color statistical information, wherein the second moment is a moment before the first moment and closest to the first moment among the image acquisition moments of the target lens module; In a case where the image acquisition moment of the target lens module includes the first moment, color statistical information of the third image acquired by the target lens module at the first moment is determined as the first color statistical information.

5. The method according to claim 4, characterized in that The obtaining of first color statistical information of a target lens module of the electronic device at a first moment includes: When the image acquisition moment of the target lens module includes the first moment, obtaining first color statistical information of the target lens module of the electronic device at the first moment; After respectively determining the image acquisition time of the first lens module and the image acquisition time of the target lens module by using the first operating frequency and the second operating frequency, the method further includes: When the image acquisition moment of the target lens module does not include the first moment, performing white balance processing on the first image acquired by the first lens module of the electronic device at the first moment using the second color compensation information; The second color compensation information is color compensation information of the first lens module at a third moment, and the third moment is a moment before the first moment in the image acquisition moments of the first lens module.

6. The method according to claim 1, characterized in that The method further comprises: When the electronic device displays the image captured by the first lens module, adjusting the resolution of the second lens module from the first resolution to the second resolution; The second resolution is smaller than the first resolution, and the first resolution is the resolution of the second lens module when the electronic device displays an image captured by the second lens module.

7. The method according to claim 1, characterized in that The target lens module further includes a third lens module of the electronic device, wherein the field of view of the third lens module is larger than the field of view of the first lens module.

8. The method according to claim 1, characterized in that The electronic device further comprises a third lens module, wherein the field of view of the third lens module is larger than the field of view of the first lens module; Before obtaining the first color statistical information of the target lens module of the electronic device at the first moment, the method further includes: Obtaining an angle between a target component of a first hyperplane and a target component of a second hyperplane, wherein the target component includes at least one of a principal component and a secondary component; the first hyperplane is a hyperplane formed by response curves of each color channel corresponding to the second lens module; and the second hyperplane is a hyperplane formed by response curves of each color channel corresponding to the third lens module; When the included angle is less than the included angle threshold, determining the second lens module as the target lens module; When the included angle is greater than or equal to the included angle threshold, the second lens module and the third lens module are determined as the target lens module.

9. The method according to claim 1, characterized in that When the target lens module is the second lens module, before performing white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information, the method further includes: When the color temperature set includes the ambient color temperature of the electronic device, determining a target compensation weight corresponding to the second lens module using a reference compensation weight corresponding to the second lens module, wherein the target compensation weight is less than the reference compensation weight; the color temperature set includes at least one color temperature, and for each color temperature of the at least one color temperature, statistical information of the second lens module and the first lens module exhibits metamerism at the color temperature; The performing white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information includes: multiplying the first color compensation information and the target compensation weight to obtain target color compensation information; Based on the target color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

10. An image processing device, characterized in that: Applied to electronic equipment, the electronic equipment includes a first lens module and a second lens module, wherein the field of view of the first lens module is smaller than the field of view of the second lens module; the device includes: a first acquisition module, configured to acquire, when the electronic device displays an image captured by the first lens module, first color statistical information of a target lens module of the electronic device at a first moment, wherein the target lens module includes the second lens module; A first determining module is configured to determine first color compensation information mapped to the first color statistical information based on a first color mapping relationship, wherein the first color mapping relationship is a color mapping relationship of the electronic device, the color mapping relationship is a mapping relationship between the color statistical information of the target lens module and the color compensation information of the first lens module; and the first color compensation information is color compensation information of the first lens module of the electronic device at the first moment; The first white balance processing module is configured to perform white balance processing on a first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information.

11. An electronic device, characterized in that: The image processing method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 9 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 9 are implemented.

Images