Image processing method and apparatus, and electronic device and readable storage medium

Abstract

The present application belongs to the technical field of image processing. Disclosed are an image processing method and apparatus, and an electronic device and a readable storage medium. The image processing method is executed by an electronic device, the electronic device comprising a first lens module and a second lens module, wherein the field-of-view of the first lens module is smaller than the field-of-view of the second lens module. The image processing method comprises: when an electronic device displays an image collected by means of a first lens module, acquiring first color statistical information of a target lens module of the electronic device at a first moment, wherein the target lens module includes a second lens module; on the basis a first color mapping relationship, determining first color compensation information mapped by the first color statistical information; and on the basis of the first color compensation information, performing white balance processing on a first image collected by means of the first lens module of the electronic device at the first moment.

Description

Image processing methods, apparatus, electronic devices and readable storage media

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202410994017.7, filed on July 23, 2024, entitled “Image Processing Method, Apparatus, Electronic Device and Readable Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of image processing technology, specifically relating to an image processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0004] In related technologies, electronic devices can integrate multiple lens modules, such as ultra-wide-angle lens modules, wide-angle lens modules, and telephoto lens modules, to meet different shooting needs.

[0005] Different lens modules have different fields of view (FOV). For lens modules with a small FOV, such as telephoto lens modules, the limited FOV can easily lead to insufficient gray areas in the color statistics, resulting in color cast in the Auto White Balance (AWB) and poor white balance processing. Summary of the Invention

[0006] The purpose of this application is to provide an image processing method, apparatus, electronic device, and readable storage medium that can solve the problem of poor white balance processing effect.

[0007] Firstly, embodiments of this application provide an image processing method.

[0008] Performed by an electronic device, the electronic device comprising a first lens module and a second lens module, wherein the field of view of the first lens module is smaller than the field of view of the second lens module; the image processing method includes:

[0009] When the electronic device displays an image captured by the first lens module, the first color statistics of the target lens module of the electronic device at a first moment are obtained, wherein the target lens module includes the second lens module;

[0010] Based on the first color mapping relationship, the first color compensation information mapped by the first color statistics information is determined, wherein the first color mapping relationship is the color mapping relationship of the electronic device, and the color mapping relationship is the mapping relationship between the color statistics information of the target lens module and the color compensation information of the first lens module; the first color compensation information is the color compensation information of the first lens module of the electronic device at the first moment.

[0011] Based on the first color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

[0012] Secondly, embodiments of this application provide an image processing apparatus applied to an electronic device, the electronic device including a first lens module and a second lens module, wherein the field of view of the first lens module is smaller than the field of view of the second lens module; the image processing apparatus includes:

[0013] The first acquisition module is used to acquire first color statistics information of the target lens module of the electronic device at a first moment when the electronic device displays an image captured by the first lens module, wherein the target lens module includes the second lens module.

[0014] The first determining module is configured to determine the first color compensation information mapped by the first color statistics information according to the first color mapping relationship, wherein the first color mapping relationship is the color mapping relationship of the electronic device, and the color mapping relationship is the mapping relationship between the color statistics information of the target lens module and the color compensation information of the first lens module; the first color compensation information is the color compensation information of the first lens module of the electronic device at the first moment.

[0015] The first white balance processing module is used to perform white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information.

[0016] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the image processing method as described in the first aspect.

[0017] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0018] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the image processing method as described in the first aspect.

[0019] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the image processing method as described in the first aspect.

[0020] In this embodiment, when the electronic device displays an image captured by the first lens module of a small FOV, the first color statistics information of the target lens module of a large FOV at the first moment can be obtained. Then, based on the mapping relationship between the color statistics information of the target lens module of the electronic device and the color compensation information of the first lens module, the first color compensation information corresponding to the first color statistics information is determined, i.e., the color compensation information of the first lens module at the first moment is obtained. Subsequently, white balance processing is performed on the first image captured by the first camera module of the electronic device at the first moment based on the first color compensation information. This embodiment, through the mapping relationship between the color statistics information of the target lens module of the electronic device and the color compensation information of the first lens module, can utilize the color statistics information of the target lens module of a large FOV to accurately determine the color compensation information of the first lens module of a small FOV. This allows for accurate color compensation of the lens module of a small FOV using the color statistics information of the target lens module of a large FOV. Since the target lens module of a large FOV can acquire more color statistics information, the white balance processing effect of the lens module of a small FOV can be improved. Attached Figure Description

[0021] Figure 1a is one of the schematic diagrams of color mapping provided in the embodiments of this application;

[0022] Figure 1b is a schematic diagram showing the relationship between the light source and AWB gain provided in the embodiments of this application;

[0023] Figure 2 is a flowchart of the image processing method provided in an embodiment of this application;

[0024] Figure 3 is a schematic diagram of image selection for color mapping provided in an embodiment of this application;

[0025] Figure 4a is a second schematic diagram of color mapping provided in an embodiment of this application;

[0026] Figure 4b is a third schematic diagram of color mapping provided in the embodiments of this application;

[0027] Figure 4c is a schematic diagram of gray area selection provided in an embodiment of this application;

[0028] Figure 5 is a schematic diagram of obtaining the calibration coefficients provided in an embodiment of this application;

[0029] Figure 6 is a schematic diagram of obtaining the prediction function provided in an embodiment of this application;

[0030] Figure 7 is a schematic diagram of obtaining AWB Gain provided in an embodiment of this application;

[0031] Figure 8a is a schematic diagram of the color gamut before two-dimensional color mapping provided in an embodiment of this application;

[0032] Figure 8b is a schematic diagram of the color gamut after two-dimensional space color mapping provided in an embodiment of this application;

[0033] Figure 9 is a structural diagram of the image processing apparatus provided in an embodiment of this application;

[0034] Figure 10 is a structural diagram of an electronic device provided in an embodiment of this application;

[0035] Figure 11 is a second structural diagram of the electronic device provided in the embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0037] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0038] For ease of understanding, some aspects of the embodiments of this application will be described below.

[0039] Multi-camera Synchronization System (MCSS): This is a hardware synchronization method that combines the existing read time with the unique low frame rate and low power consumption characteristics of MCSS.

[0040] Spatial Alignment Transform (SAT): Enables automatic switching between different lens modules during continuous zoom in preview or recording, and reduces problems such as image misalignment and distortion caused by switching.

[0041] Color mapping: Based on the color information of the lens module before switching, combined with the algorithm model, the color information of the lens module after switching can be predicted.

[0042] Auto White Balance (AWB) refers to the process in digital cameras or mobile phones of measuring and adjusting white objects in a scene to ensure that the white parts in the captured image achieve a realistic and natural color, while other colors are also accurately reproduced.

[0043] One-Time Programmable (OTP) data or programs are color-related characteristic parameters of the lens module, such as gray point response results at various color temperatures.

[0044] Golden: Statistics on the entire batch of lens modules show that their OTP parameters exhibit a normal distribution as the number of samples increases. In this application embodiment, samples whose OTP parameters conform to the mean of the normal distribution are called Golden, indicating that their color characteristics best represent the statistical regularity of the entire batch of lens modules.

[0045] With the development of mobile imaging in the mobile phone industry, mainstream manufacturers are no longer just focusing on piling up hardware specifications and functional parameters, but are paying more and more attention to the user's basic experience, such as how to ensure the consistency of the effect during the switching of SAT lens modules and the color accuracy of small FOV lens modules (such as telephoto lens modules). These experience optimizations require engineers to spend more time thinking, breaking down the technology from different dimensions and designing and deploying solutions at the system level.

[0046] To address the aforementioned issues, the color information of the lens module after switching can be mapped based on the color information of the lens module before switching, thereby improving color consistency before and after the lens module switch. Simultaneously, when the small FOV lens module is displayed, the larger FOV lens module's ability to obtain more statistical information can be leveraged to assist the small FOV lens module in performing real-time color calibration (AWB).

[0047] In some embodiments, to improve the color accuracy of the small FOV lens module, the large FOV lens module can be kept always on, and the statistical results can be mapped to the small FOV lens module. As shown in Figure 1a, taking the zoom from the wide-angle lens module to the telephoto lens module as an example: during the 2.8x-3.3x zoom range, both the wide-angle and telephoto lens modules are enabled simultaneously to ensure color consistency before and after the lens module switch. However, during the zoom range beyond 3.3x, the wide-angle lens module (large FOV) remains enabled to ensure the color accuracy of the telephoto lens module (small FOV). Therefore, this dual-lens-module strategy increases the power consumption of the mobile phone.

[0048] The main principle of the color mapping scheme is to map the AWB gain based on the prior information of multiple lens modules. Theoretically, it requires finding a one-to-one mapping function for the light source, the AWB gain of the large FOV lens module (e.g., AWB Gain_a, AWB Gain_b, ..., AWB Gain_n), and the AWB gain of the small FOV lens module (e.g., AWB Gain_A, AWB Gain_B, ..., AWB Gain_N). However, due to the metamerism between the integral results of the ambient reflectance spectrum and the lens module response curve, it cannot be guaranteed that these N will maintain a one-to-one mapping relationship under all reflectance spectra. In other words, there is a probability that the AWB gain of the large FOV lens module will be different under different light sources, but after mapping, the AWB gain of the small FOV lens module will be the same or very close. This will lead to color cast in the small FOV lens module. For easier understanding, please refer to Figure 1b.

[0049] Based on this, this application proposes a novel color mapping scheme by improving the MCSS hard synchronization system. This scheme can significantly optimize power consumption increments while solving the color jump problem caused by lens module switching during SAT, and improve the color accuracy of small FOV lens modules.

[0050] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0051] The image processing method of this application embodiment can be executed by an electronic device, or applied to an electronic device. The electronic device of this application embodiment may include, but is not limited to, a first lens module and a second lens module, wherein the field of view of the first lens module is smaller than the field of view of the second lens module. Further, the electronic device may also include a third lens module, the field of view of the third lens module being larger than the field of view of the first lens module. Additionally, the field of view of the third lens module may be smaller than the field of view of the second lens module, or larger than the field of view of the second lens module.

[0052] The embodiments of this application do not limit the specific forms of the first lens module, the second lens module, and the third lens module. In some embodiments, the first lens module can be a telephoto lens module, the second lens module can be one of an ultra-wide-angle lens module and a wide-angle lens module, and the third lens module can be the other of an ultra-wide-angle lens module and a wide-angle lens module. In other embodiments, the first lens module can be a wide-angle lens module, and the second lens module can be an ultra-wide-angle lens module.

[0053] As shown in Figure 2, the image processing method of this application embodiment may include:

[0054] Step 201: When the electronic device displays the image captured by the first lens module, obtain the first color statistics information of the target lens module of the electronic device at a first moment, wherein the target lens module includes the second lens module.

[0055] In this embodiment of the application, when the electronic device displays an image captured by the first lens module, the electronic device can be considered to be in the display stage of the first lens module.

[0056] During the display phase of the first lens module, the target lens module of the electronic device can be in an active state to perform color compensation on the first lens module and improve its white balance processing effect. Meanwhile, other lens modules of the electronic device, besides the first and target lens modules, can be in a switched-off state during the display phase of the first lens module, thus reducing the power consumption of the electronic device.

[0057] In practice, the target lens module may include some or all of the lens modules in the electronic device with a FOV smaller than that of the first lens module. Therefore, the target lens module can be called a lens module with a large FOV, and the first lens module can be called a lens module with a small FOV.

[0058] In some embodiments, the target lens module can be in an always-on state. In this state, the target lens module can operate at its default frequency and resolution.

[0059] In other embodiments, the target lens module can operate in at least one of a low-power (LowFps) state and a low-resolution (LowRes) state. In the low-power state, the operating frequency of the target lens module can be 1 / k of the default operating frequency, where k is a power of 2 and greater than 1, such as k being 4, 16, 256, etc. In the low-resolution state, the resolution of the target lens module can be 1 / x of the default resolution, where x is an integer greater than 1, such as x being 2, 3, 4, 5, etc. It is evident that the electronic device consumes less power in the low-power and low-resolution states compared to the normally on state.

[0060] The number of target lens modules can be greater than or equal to 1. When the number of target lens modules is greater than 1, the working states of each target lens module can be the same or different, which can be determined according to actual needs. This application embodiment does not limit this.

[0061] In this embodiment, since the target lens module of the large FOV can obtain more color statistics information, the color statistics information of the target lens module can be used to perform color compensation on the first lens module of the small FOV. In this way, the white balance processing effect of the lens module of the small FOV can be improved.

[0062] For ease of understanding, this application uses color compensation of the first lens module at a first moment as an example. The first moment refers to any image acquisition moment of the first lens module during the display phase. In other words, for any image acquired by the first lens module at any moment, the image processing method of this application embodiment can be used to perform white balance processing.

[0063] In practice, the first color statistics of the target lens module of the electronic device at the first moment can be obtained.

[0064] In this embodiment, the first color statistics of a target lens module at a first moment can be understood as: the color statistics of the target image acquired by the target lens module. The target image can be: a third image acquired by the target lens module at the first moment, or a second image acquired by the target lens module at its last historical image acquisition moment, or a copy of the second image. The last historical image acquisition moment of the target lens module, i.e., the moment among the image acquisition moments of the target lens module that is before the first moment and closest to the first moment, can be referred to as the second moment.

[0065] In some embodiments, the target image captured by the target lens module can be predetermined.

[0066] In other embodiments, the specific appearance of the target image acquired by the target lens module can be determined based on the determination result of whether the first moment is the image acquisition moment of the target lens module.

[0067] It should be noted that in practical applications, the operating frequency of each target lens module may be the same as or different from that of the first lens module.

[0068] For a target lens module whose operating frequency is the same as that of the first lens module, its image acquisition time is the same as that of the first lens module. In this case, the first moment is also the image acquisition time of the target lens module, that is, the target lens module will perform image acquisition at the first moment.

[0069] For a target lens module whose operating frequency differs from that of the first lens module, its image acquisition time will differ from that of the first lens module. In this case, the first moment may or may not be the image acquisition time of the target lens module.

[0070] If the first moment is the image acquisition moment of a certain target lens module, that is, the image acquisition moment of the target lens module includes the first moment, then the first color statistics of the target lens module at the first moment can be understood as: the color statistics of the third image acquired by the target lens module at the first moment.

[0071] If the first moment is the image acquisition moment of a target lens module, that is, the image acquisition moment of the target lens module includes the first moment, then the first color statistics of the target lens module at the first moment can be understood as: the color statistics of the second image acquired by the target lens module at the second moment, or the color statistics of the copy image of the third image of the target lens module.

[0072] This application does not limit the specific form of color statistics information. In some embodiments, color statistics information may include Gr information and Gb information. Methods for obtaining color statistics information can be found in related technologies and will not be described here.

[0073] Step 202: Determine the first color compensation information mapped by the first color statistics information according to the first color mapping relationship, wherein the first color mapping relationship is the color mapping relationship of the electronic device, and the color mapping relationship is the mapping relationship between the color statistics information of the target lens module and the color compensation information of the first lens module; the first color compensation information is the color compensation information of the first lens module of the electronic device at the first moment.

[0074] In this embodiment, the electronic device stores its own color mapping relationship, referred to as the first color mapping relationship. The first color mapping relationship can be used to describe the mapping relationship between the color statistics information of the target lens module of the electronic device and the color compensation information of the first lens module of the electronic device.

[0075] The embodiments of this application do not limit the representation method of color mapping relationship. Color mapping relationship can be represented by functions, network models, mapping tables, etc.

[0076] When the color mapping relationship is represented by a function, the independent variable of the function is the color statistics of the target lens module, and the dependent variable is the color compensation information of the first lens module. In this case, the color compensation information of the first lens module of the electronic device at the first moment can be obtained by substituting the first color statistics into the function. This function can be called a prediction function.

[0077] When the color mapping relationship is represented by a network model, the input to the network model is the color statistics of the target lens module, and the output of the network model is the color compensation information of the first lens module. In this case, the color compensation information of the first lens module of the electronic device at the first moment can be obtained by inputting the first color statistics into the network model.

[0078] When the color mapping relationship is represented by a mapping table, the color compensation information of the first lens module of the electronic device at the first moment can be obtained by looking up the first color statistics in the mapping table.

[0079] The embodiments of this application do not limit the method of obtaining the first color mapping relationship. In some embodiments, the first color mapping relationship can be determined based on a reference color mapping relationship, as detailed in the following description, which will not be repeated here. In other embodiments, the first color mapping relationship can be generated based on test data of the electronic device, and its generation method is similar to that of the reference color mapping relationship, which will not be repeated here. In still other embodiments, the first color mapping relationship can be pre-written into the electronic device.

[0080] Step 203: Based on the first color compensation information, perform white balance processing on the first image captured by the first lens module of the electronic device at the first moment.

[0081] In some embodiments, the AWB gain of the first lens module at a first moment can be determined directly using the first color compensation information, such as by taking the reciprocal of the first color compensation information to obtain the AWB gain, and then using the AWB gain of the first lens module at the first moment to perform white balance processing on the first image.

[0082] In other embodiments, white balance processing of the first image can be performed by combining the first color compensation information and the color statistics information of the first lens module of the electronic device at a first moment. Specifically, a weighted average can be calculated between the first color compensation information and the color statistics information of the first lens module of the electronic device at the first moment to obtain the target color statistics information of the first lens module at the first moment. Then, the AWB gain of the first lens module at the first moment can be determined using the target color statistics information, such as by taking the reciprocal of the target color statistics information. This AWB gain is then used to perform white balance processing on the first image. In this embodiment, the AWB gain of the first lens module at the first moment is determined based on the color compensation information and its own color statistics information. Therefore, compared to directly using the color compensation information to determine the AWB gain, the white balance processing effect can be further improved.

[0083] The image processing method of this application embodiment, when an electronic device displays an image captured by a first lens module with a small field of view (FOV), can obtain the first color statistics information of a target lens module with a large FOV at a first moment. Then, based on the mapping relationship between the color statistics information of the target lens module of the electronic device and the color compensation information of the first lens module, the first color compensation information corresponding to the first color statistics information is determined, i.e., the color compensation information of the first lens module at the first moment is obtained. Subsequently, white balance processing is performed on the first image captured by the first camera module of the electronic device at the first moment based on the first color compensation information. This application embodiment, through the mapping relationship between the color statistics information of the target lens module of the electronic device and the color compensation information of the first lens module, can utilize the color statistics information of the target lens module with a large FOV to accurately determine the color compensation information of the first lens module with a small FOV. This allows for accurate color compensation of the lens module with a small FOV using the color statistics information of the target lens module with a large FOV. Since the target lens module with a large FOV can acquire more color statistics information, the white balance processing effect of the lens module with a small FOV can be improved.

[0084] The following provides a detailed explanation of how to obtain the first color mapping relationship for electronic devices.

[0085] In some embodiments, before determining the first color compensation information mapped from the first color statistics information according to the first color mapping relationship, the image processing method further includes:

[0086] Using the color statistics information of the electronic device and the color statistics information of the reference device, the color calibration coefficient of the electronic device relative to the reference device is determined;

[0087] The first color mapping relationship is determined using the color calibration coefficient and the reference color mapping relationship, wherein the reference color mapping relationship is the color mapping relationship of the reference device.

[0088] In this embodiment, the first color mapping relationship is determined based on a reference color mapping relationship. The reference color mapping relationship is the color mapping relationship of a reference device, used to describe the mapping relationship between the color statistics information of the target lens module of the reference device and the color compensation information of the first lens module of the reference device.

[0089] The electronic device and the reference device include the same lens module, and the lens modules in both are from the same batch. The OTP parameters of the lens modules in the same batch follow a normal distribution. The OTP parameters of a lens module refer to the color-related characteristic parameters of the lens module, such as the gray point response results at various color temperatures and color statistics. In this application embodiment, the lens module whose OTP parameters conform to the mean of a normal distribution is called the "Golden" lens module, indicating that its color characteristics best represent the statistical regularity of the entire batch of lens modules. The electronic device including the Golden lens module is called the reference device.

[0090] In practice, the color statistics of the electronic device and the reference device can be used to determine the color calibration coefficient of the electronic device relative to the reference device. Then, the color calibration coefficient is multiplied into the reference color mapping relationship to obtain the first color mapping relationship.

[0091] It should be noted that the color statistics information used to determine the calibration coefficients is the color statistics information obtained before the equipment leaves the factory.

[0092] In some embodiments, the color mapping relationship may include the mapping relationship between the color statistics of the target lens module and the color compensation information of the first lens module under each color temperature category. In this embodiment, the color calibration coefficient of the electronic device relative to the reference device may include the color calibration coefficient of the electronic device relative to the reference device under each color temperature category. The color statistics of the electronic device include the color statistics of the electronic device under each color temperature category, and the color statistics of the reference device include the color statistics of the reference device under each color temperature category.

[0093] This application does not limit the method of color temperature classification. In some embodiments, color temperature classification may include, but is not limited to, high color temperature, interpolated color temperature, and low color temperature. Further, high color temperature may include D75, D65, and D50; interpolated color temperature may include TL84 and CWF; and low color temperature may include H, A, and U30. In this embodiment, the color temperature range of the entire natural world can be sampled according to eight spectral groups: H, A, U30, TL84, CWF, D50, D65, and D75. The distribution pattern of the lens module's gray area response under different light sources is statistically analyzed to obtain the color statistical information of the lens module under each color temperature classification.

[0094] In determining the color calibration coefficient of an electronic device relative to a reference device, in some embodiments, the color calibration coefficient corresponding to the same lens module of the electronic device and the reference device can be calculated using the color statistical information of the same lens module, and used as the color calibration coefficient of the electronic device relative to the reference device; in other embodiments, the color calibration coefficient corresponding to each lens module can be calculated first, and then the color calibration coefficient corresponding to each lens module can be used to determine the color calibration coefficient of the electronic device relative to the reference device.

[0095] The color calibration coefficient corresponding to the lens module can be obtained by dividing the color statistics of the lens module of the electronic device by the color statistics of the lens module of the reference device.

[0096] In this embodiment, the first color mapping relationship is determined based on a reference color mapping relationship. Since the color characteristics of the lens module of the reference device can represent the statistical regularity of the entire batch of lens modules, the color mapping relationship of the electronic device is determined based on the color mapping relationship of the reference device. In this way, the color mapping accuracy of the color mapping relationship of the electronic device can be improved.

[0097] The following provides a detailed explanation of how to obtain the reference color mapping relationship.

[0098] In some embodiments, before determining the first color mapping relationship using the color calibration coefficient and the reference color mapping relationship, the image processing method may further include:

[0099] A first matrix is ​​constructed using the color statistics information of the target lens module of the reference device;

[0100] Using the color statistics information of the first lens module of the reference device, a second matrix is ​​constructed, wherein the dimension of the first matrix is ​​greater than or equal to the dimension of the second matrix;

[0101] Establish the initial color mapping relationship;

[0102] Based on the first matrix and the second matrix, the initial color mapping relationship is trained to obtain the reference color mapping relationship.

[0103] In this embodiment, the input and output formats of the initial color mapping relationship are both matrices, but this does not limit the input and output formats of the initial color mapping relationship. In other embodiments, the input and output formats of the initial color mapping relationship can be vectors, etc.

[0104] In practical implementation, the color statistics of the target lens modules of the reference device can be used to construct a first matrix. When the number of target lens modules is one, the first matrix includes all color statistics of the target lens modules of the reference device; when the number of target lens modules is greater than one, the first matrix, in addition to including the color statistics of all target lens modules, may include the complete color statistics of at least one target lens module of the reference device. That is, when the number of target lens modules is greater than one, for some target lens modules, the first matrix may only include a portion of their color statistics.

[0105] For ease of understanding, the following example is provided:

[0106] Assuming the color statistics include Gr and Gb, and the target lens modules include an ultra-wide-angle lens module (hereinafter referred to as UW) and a wide-angle lens module (hereinafter referred to as W), then the first matrix can include Gr+Gb of UW and Gr+Gb of W; or, Gr+Gb of UW and Gr of W; or, Gr+Gb of UW and Gb of W; or, Gr of UW and Gr+Gb of W; or, Gb of UW and Gr+Gb of W.

[0107] A second matrix can be constructed using the color statistics information of the first lens module of the reference device. The first matrix includes all the color statistics information of the first lens module of the reference device.

[0108] It is understandable that when the number of target lens modules is 1, the dimensions of the first matrix and the second matrix are equal.

[0109] When the number of target lens modules is greater than one, the dimension of the first matrix is ​​greater than the dimension of the second matrix. In this case, the reference color mapping relationship obtained through training can map low-dimensional color compensation information to high-dimensional color statistics, thus reducing the cases of linear inseparability and improving the accuracy of color mapping.

[0110] Alternatively, an initial color mapping relationship can be constructed. Then, the first matrix is ​​used as the input to the initial color mapping relationship, and the second matrix is ​​used as the calibration output of the initial color mapping relationship. The initial color mapping relationship is trained until the training stopping condition is met, resulting in a reference color mapping relationship. This application does not limit the specific form of the training stopping condition. In some embodiments, the training stopping condition can be expressed as: the norm of the matrix output by the initial color mapping relationship is minimized with that of the second matrix, that is, the matrix output by the initial color mapping relationship is close to the second matrix.

[0111] In this embodiment, an initial color mapping relationship can be pre-constructed. Then, based on the first matrix constructed using the color statistics information of the target lens module of the reference device, and the color statistics information of the first lens module of the reference device, a second matrix is ​​constructed. The initial mapping relationship is trained to obtain the reference color mapping relationship. In this way, the reliability of the reference color mapping relationship can be improved.

[0112] In other embodiments, the reference color mapping relationship can be obtained by regression fitting of the color statistics of the target lens module of the reference device and the color statistics of the first lens module of the reference device, thus simplifying the acquisition of the reference color mapping relationship.

[0113] The following provides a detailed explanation of the first color statistics of the target lens module of the electronic device at the first moment.

[0114] In some embodiments, before obtaining the first color statistics information of the target lens module of the electronic device at a first moment, the method further includes:

[0115] When the electronic device displays the image captured by the first lens module, the target lens module is adjusted to a second operating frequency, wherein the second operating frequency is less than the first operating frequency, and the first operating frequency is the operating frequency of the first lens module.

[0116] Using the first operating frequency and the second operating frequency, the image acquisition time of the first lens module and the image acquisition time of the target lens module are determined respectively.

[0117] The acquisition of the first color statistics information of the target lens module of the electronic device at a first moment includes at least one of the following:

[0118] If the image acquisition time of the target lens module does not include the first time, the color statistics information of the second image acquired by the target lens module at the second time is determined as the first color statistics information, wherein the second time is the time in the image acquisition time of the target lens module that is before the first time and is closest to the first time.

[0119] If the image acquisition time of the target lens module includes the first time, the color statistics information of the third image acquired by the target lens module at the first time is determined as the first color statistics information.

[0120] In this embodiment, during the display stage of the first lens module, the target lens module is mainly used to perform color compensation on the first lens module. It can control the target lens module to operate at a low frequency to reduce the hardware power consumption of the electronic device.

[0121] In practice, the operating frequency of the target lens module can be adjusted based on the operating frequency of the first lens module, so that the operating frequency of the target lens module is lower than that of the first lens module. It should be noted that when the number of target lens modules is greater than one, the operating frequencies of the different target lens modules can be the same or different, and can be set according to the actual situation. This application embodiment does not limit this.

[0122] Once the operating frequency of the lens module is determined, its image acquisition time is also determined accordingly. It can be understood that if the operating frequency of the target lens module is lower than that of the first lens module, the image acquisition time of the target lens module will be shorter than that of the first lens module. Therefore, the first moment may or may not be the image acquisition time of the target lens module.

[0123] When the first moment is the image acquisition moment of the target lens module, the first color statistics of the target lens module at that moment are the same as the color statistics of the third image acquired by the target lens module at that moment. In other words, in this case, color compensation for the first lens module at the first moment is performed using the color statistics of the target lens module at that moment. Since the third image was acquired at the same time as the first image, this improves the accuracy of color compensation for the first lens module.

[0124] If the first moment is not the image acquisition moment of the target lens module, the first color statistics of the target lens module at the first moment are the same as the color statistics of the third image acquired by the target lens module at the second moment. In other words, in this case, the color of the first lens module at the first moment is compensated by the color statistics of the target lens module at the historical image acquisition moment closest to the first moment. Since the acquisition time of the last image acquired by the target lens module is close to the first moment, the accuracy of the color compensation of the first lens module can be improved.

[0125] By employing the above method, when performing color compensation on the first lens module at the first moment, regardless of whether the first moment is the image acquisition time of the target lens module, the image acquired by the target lens module can be used for color compensation of the first lens module. This improves the reliability of color compensation for the first lens module. Furthermore, if the first moment is the image acquisition time of the target lens module, color compensation is performed on the first lens module using the color statistics of the third image acquired by the target lens module at the first moment. Since the third image was acquired at the same time as the first image, this improves the accuracy of color compensation for the first lens module. If the first moment is not the image acquisition time of the target lens module, color compensation is performed on the first lens module using the color statistics of the last image acquired by the target lens module. Since the acquisition time of the last image acquired by the target lens module is close to the first moment, this improves the accuracy of color compensation for the first lens module.

[0126] In some embodiments, obtaining the first color statistics information of the target lens module of the electronic device at a first moment includes:

[0127] When the image acquisition time of the target lens module includes the first time, the first color statistics information of the target lens module of the electronic device at the first time is obtained;

[0128] After determining the image acquisition time of the first lens module and the image acquisition time of the target lens module using the first operating frequency and the second operating frequency respectively, the method further includes:

[0129] If the image acquisition time of the target lens module does not include the first time, the first image acquired by the first lens module of the electronic device at the first time is processed for white balance using the second color compensation information.

[0130] Wherein, the second color compensation information is the color compensation information of the first lens module at the third moment, and the third moment is the moment in the image acquisition time of the first lens module that is before the first moment.

[0131] It should be noted that when the number of target lens modules is greater than 1, as long as the image acquisition time of any one target lens module includes the first moment, the image acquisition time of the target lens modules of the visual electronic device can be considered to include the first moment; only when the image acquisition times of all target lens modules do not include the first moment can the image acquisition time of the target lens modules of the visual electronic device not include the first moment.

[0132] In this embodiment, if the image acquisition time of the target lens module of the electronic device includes the first moment, the first image can be white-balanced through steps 101 to 103.

[0133] If the image acquisition time of the target lens module of the electronic device does not include the first moment, and white balance processing is performed on the first image through steps 101 to 103, since the first color statistics information of the target lens module of the electronic device at the first moment is equivalent to the color statistics information of the second image acquired by the target lens module of the electronic device at the second moment, the calculated first color compensation information is the same as the second color compensation information calculated using the second color statistics information of the target lens module at the second moment. Based on this, in this case, the second color compensation information can be directly reused to perform white balance processing on the first image. This reduces the number of calculations for color compensation information, thereby further reducing the operating power consumption of the electronic device.

[0134] For ease of understanding, the following example is provided:

[0135] Assuming the first lens module is a telephoto lens module, the target lens module includes an ultra-wide-angle lens module and a wide-angle lens module. The operating frequency of the wide-angle lens module is 1 / 2 that of the telephoto lens module, and the operating frequency of the ultra-wide-angle lens module is 1 / 3 that of the telephoto lens module.

[0136] The image selection strategy for color compensation of the first lens module is shown in Figure 3. In Figure 3, blank boxes represent images captured by the lens module, and filled boxes represent copies of the last captured image. Based on the correspondence of the operating frequencies of the three camera modules, as shown in Figure 3, the telephoto lens module captures 2 images, and the wide-angle lens module captures 1 image; the telephoto lens module captures 3 images, and the ultra-wide-angle lens module captures 1 image.

[0137] When performing white balance compensation on the first frame image captured by the telephoto lens module, the color compensation information of the first frame image captured by the telephoto lens module can be calculated using the color statistics of the first frame image captured by the wide-angle lens module and the first frame image captured by the ultra-wide-angle lens module.

[0138] When performing white balance compensation on the second frame image captured by the telephoto lens module, since the image capture time is not the same as the image capture time of the wide-angle lens module and the ultra-wide-angle lens module, the color compensation information of the second frame image captured by the telephoto lens module can be calculated using the color statistics information of the first frame image captured by the wide-angle lens module and the first frame image captured by the ultra-wide-angle lens module.

[0139] It can be seen that the color compensation information of the first frame image and the second frame image captured by the telephoto lens module is the same. Therefore, the color compensation information of the first frame image captured by the telephoto lens module can be directly reused to perform white balance processing on the second frame image captured by the telephoto lens module.

[0140] In this embodiment, when the image acquisition time of the target lens module of the electronic device includes the first moment, the color compensation information of the first image can be obtained by calculation; when the image acquisition time of the target lens module of the electronic device does not include the first moment, the color compensation information obtained by the last calculation can be directly reused as the color compensation information of the first image. In this way, the number of calculations of color compensation information can be reduced, thereby further reducing the operating power consumption of the electronic device.

[0141] In some embodiments of this application, the image processing method may further include:

[0142] When the electronic device displays an image captured by the first lens module, the resolution of the second lens module is adjusted from the first resolution to the second resolution;

[0143] Wherein, the second resolution is less than the first resolution, and the first resolution is the resolution of the second lens module during the process of the electronic device displaying the image captured by the second lens module.

[0144] In this embodiment, considering that the resolution required for white balance processing is not high, the target lens module can be controlled to operate in a low-resolution state during the display stage of the first lens module, thereby reducing the hardware power consumption of the electronic device.

[0145] The determination of the target lens module in the embodiments of this application will be described in detail below.

[0146] In some embodiments, the target lens module further includes a third lens module of the electronic device, wherein the field of view of the third lens module is larger than the field of view of the first lens module.

[0147] In this embodiment, the part or all of the lens modules of the electronic device whose FOV is greater than that of the first lens module can be directly identified as the target lens module. This simplifies the method of determining the target lens module and improves the efficiency of determining the target lens module.

[0148] In some embodiments, the electronic device further includes a third lens module, the field of view of which is larger than that of the first lens module;

[0149] Before acquiring the first color statistics information of the target lens module of the electronic device at the first moment, the image processing method may further include:

[0150] Obtain the angle between the target components of the first hyperplane and the target components of the second hyperplane, wherein the target components include at least one of the principal components and the secondary components; the first hyperplane is a hyperplane formed by the response curves of each color channel corresponding to the second lens module; the second hyperplane is a hyperplane formed by the response curves of each color channel corresponding to the third lens module.

[0151] If the included angle is less than the included angle threshold, the second lens module is determined as the target lens module;

[0152] If the included angle is greater than or equal to the included angle threshold, the second lens module and the third lens module are identified as the target lens module.

[0153] In this embodiment, the electronic device includes at least two lens modules with a FOV smaller than that of the first lens module, and the target lens module can be selected from these at least two lens modules.

[0154] In practice, for each of the at least two lens modules, a hyperplane can be constructed using the response curves of its corresponding color channels. Then, for any two lens modules, at least one of the following can be calculated: the angle between the principal components of their corresponding constituent surfaces, and the angle between the secondary components of their corresponding constituent surfaces. These angles are then compared with their corresponding angle thresholds, and the target lens module is determined based on the comparison results.

[0155] Specifically, if the included angle is less than its corresponding included angle threshold, it indicates that the response functions of the two lens modules are relatively similar, and one of the two lens modules can be selected as the target lens module. If the included angle is greater than its corresponding included angle threshold, it indicates that the response functions of the two lens modules are relatively similar, and both lens modules can be identified as the target lens modules.

[0156] The included angle thresholds for the principal and secondary components can be preset. In one example, the included angle threshold for the principal component can be 2°, and the included angle threshold for the secondary component can be 5°, but this is not the only option. The color channels can be red (R), green (G), and blue (B) channels.

[0157] The target lens module determined by the above method can reduce the amount of data in the first color statistics, thereby reducing the burden of color mapping.

[0158] In some embodiments, when the target lens module is the second lens module, before performing white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information, the image processing method further includes:

[0159] When the color temperature set includes the ambient color temperature of the electronic device, the target compensation weight corresponding to the second lens module is determined using the reference compensation weight corresponding to the second lens module, wherein the target compensation weight is less than the reference compensation weight; the color temperature set includes at least one color temperature, and for each of the at least one color temperature, the statistical information of the second lens module and the first lens module has a metamerism problem at the color temperature;

[0160] The step of performing white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information includes:

[0161] Multiply the first color compensation information and the target compensation weight to obtain the target color compensation information;

[0162] Based on the target color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

[0163] In this embodiment, the number of target lens modules is 1, and there is a high probability that the statistical information of the target lens module and the first lens module will have metamerism.

[0164] First, determine whether the statistical information of the target lens module and the first lens module has metamerism under the current ambient color temperature of the electronic device.

[0165] In practice, a color temperature set can be obtained in advance. This color temperature set includes all color temperatures corresponding to the metamerism problem between the target lens module and the first lens module, based on statistical information. It can be determined, but is not limited to, through the following methods:

[0166] Obtain the mapping relationship between the sampling points corresponding to the color statistics information of the first lens module and the sampling points corresponding to the color statistics information of the target lens module;

[0167] If at least two sampling points in the first lens module are mapped to the first sampling point corresponding to the target lens module, it is determined that the color statistics of the first lens module and the color statistics of the target lens module have metamerism at the color temperature corresponding to the first sampling point.

[0168] After obtaining the color temperature set, it is possible to determine whether there is a metamerism problem in the statistical information of the target lens module and the first lens module under the current color temperature of the electronic device by determining whether the current ambient color temperature of the electronic device is within the color temperature set.

[0169] If the color temperature set includes the current ambient color temperature of the electronic device, it can be determined that the statistical information of the target lens module and the first lens module has a metamerism problem under the current ambient color temperature. When using the first color compensation information to perform white balance processing on the first image, the compensation weight corresponding to the target lens module can be reduced to reduce the contribution of the first color compensation information to the white balance processing of the first image, thereby reducing the impact of the metamerism problem on the white balance processing and improving the white balance processing effect.

[0170] If the color temperature set does not include the current ambient color temperature of the electronic device, it can be determined that the statistical information of the target lens module and the first lens module does not have the problem of metamerism under the current ambient color temperature. Therefore, the compensation weight corresponding to the target lens module does not need to be adjusted, and the contribution of the first color compensation information to the white balance processing of the first image can be used.

[0171] By using the above method, when the statistical information of the target lens module and the first lens module has metamerism under the current ambient color temperature, the contribution of the first color compensation information to the white balance processing of the first image can be reduced by decreasing the compensation weight corresponding to the target lens module, thereby reducing the impact of metamerism on white balance processing and improving the white balance processing effect.

[0172] It should be noted that the various optional embodiments described in this application can be combined with each other or implemented individually without conflict, and this application does not limit the implementation of these embodiments.

[0173] To facilitate understanding of the image processing method provided in the above embodiments, the following describes the image processing method using specific scenario embodiments.

[0174] In the following scenario embodiment, the first lens module is a telephoto lens module, and the target lens module includes at least one of an ultra-wide-angle lens module and a wide-angle lens module. Color statistics are referred to simply as statistics.

[0175] The image processing method in this scenario embodiment may include the following steps:

[0176] Step 1: Because color mapping calculations do not require a high frequency, and the resolution of the statistical information needed for AWB calculations does not need to be very high, in some embodiments, as shown in Figure 4a, the always-on phase can be replaced with a low frame rate (LowFps) & low resolution (LowRes) phase. For example, the frame rate can be compressed to 1 / x of the original (x = 2, 3, 4, 5), while the resolution can be compressed to 1 / k of the original (k = 4, 16, 256).

[0177] Step 2: Based on Step 1, a special processing strategy is applied to the telephoto lens module, as shown in Figure 4b.

[0178] Because the field of view (FOV) of telephoto lens modules is limited, there's a possibility of color cast in AWB due to insufficient gray areas in the statistical information of the telephoto lens. However, if only a wide-angle lens module is used for color mapping, there's a chance of encountering many-to-one metamerism, a problem inherent in predicting data in two-dimensional space, which makes linear inseparability more likely. Step two innovatively addresses classification and regression problems by constructing higher-dimensional data.

[0179] Because the response functions of ultra-wide-angle and wide-angle lens modules are different, the two-dimensional subspace spanned by the statistical information obtained by integrating their respective light source spectra is also linearly uncorrelated. Step two constructs a four-dimensional space using the statistical information of the wide-angle and ultra-wide-angle lens modules, upgrading the Gr and Gb information of the original wide-angle lens module to wide-angle Gr and Gb information and ultra-wide-angle Gr and Gb information, in order to solve the metamerism problem of linearly inseparable two-dimensional subspace.

[0180] The main strategy in step two is to call up both the ultra-wide-angle and wide-angle lens modules to be in the low frame rate & low resolution mode of MCSS when the telephoto lens module is displayed. The frame rate of the ultra-wide-angle lens module can be lower than that of the wide-angle lens module.

[0181] Assume the wide-angle lens module operates at half the frequency of the telephoto lens module, and the ultra-wide-angle lens module operates at one-third the frame rate of the telephoto lens module. The image selection strategy for the three lens modules is shown in Figure 3.

[0182] Step 3: The integrals of different reflectance spectra and lens module response functions exhibit a priori distribution patterns across the entire grayscale space. By photographing grayscale cards (with uniform reflectance across the 380nm-800nm ​​range) under a light source spectrum of 2000K-10000K, we can sample the entire natural color temperature range using eight spectral groups: H, A, U30, TL84, CWF, D50, D65, and D75. We can then statistically analyze the distribution patterns of each lens module's grayscale response under different light sources, obtaining statistical information for each lens module at different color temperatures. The grayscale selection can be seen within the white dashed box in Figure 4c.

[0183] Step four, following Step three, requires calibration using a Golden phone to ensure the best possible color reproduction for these lens modules. This involves preprocessing the data from both the Golden module (the lens module of the Golden phone) and the OTP module (the lens module of the electronic device) to obtain their calibration ratios. Since OTP and Golden are currently calibrated according to high color temperature, low color temperature, and interpolated color temperature, the calibration ratios need to be calculated separately for each color temperature. D75, D65, and D50 can be classified as high color temperature, TL84 and CWF as interpolated color temperature, and H, A, and U30 as low color temperature. See Figure 5 for the detailed approach.

[0184] Step 5: Based on steps 2 to 4, construct the four-dimensional input matrix (inputMatrix) and the two-dimensional output matrix (outputMatrix), as follows:

[0185] Input matrix:

[0186] Output matrix:

[0187] The parameters in the above matrix are all statistical values ​​from the Golden phone. uw This refers to the Gr value of the ultra-wide-angle lens module on the Golden phone; Gr w This refers to the Gr value of the wide-angle lens module on the Golden phone; Gr tele This indicates the Gr value of the telephoto lens module on the Golden phone.

[0188] Step Six: Construct a prediction function (i.e., a color mapping relationship) and perform regression fitting on the input and output matrices from Step Five. Ensure that the L2 norm of the result of the input matrix after passing through the prediction function is minimized with respect to the output matrix. This yields the prediction matrix PreFunc. The process can be described by the following formula; the flowchart is shown in Figure 6:

[0189] argmin||PreFunc(inputMatrix)-outputMatrix|| 2

[0190] Step 7, the deployment mapping process, involves writing the prediction function PreFunc into the program and multiplying the ratio of the corresponding effective mobile phone (electronic device) into the prediction function to obtain the prediction function of this mobile phone itself.

[0191] Step 8, the mapping process, involves feeding the statistical information from the ultra-wide-angle and wide-angle lens modules into Step 7 to obtain the color prediction value for the telephoto lens module. This color prediction value is then used to determine the AWB Gain. See Figure 7 for a detailed implementation.

[0192] Step 9: If the measured difference in response function between different lens modules meets a certain statistical range, that is, the angle between the principal components of the hyperplane of different lens modules is less than the corresponding angle threshold, or the angle between the secondary principal components of the hyperplane of different lens modules is less than the corresponding angle threshold, the input matrix in Step 5 can be simplified. For example, the statistical information of the ultra-wide-angle or wide-angle lens modules can be used to perform color mapping on the telephoto lens module. The rest still needs to meet the steps in Steps 3 to 8.

[0193] Figures 8a and 8b illustrate the distribution of gray areas before and after color mapping in two-dimensional space. Figure 8a shows the distribution of gray areas for ultra-wide-angle or wide-angle lens modules, while Figure 8b shows the distribution of gray areas for telephoto lens modules. The horizontal and vertical axes are Gr and Gb, respectively, and the black dots represent the sampling coordinates of gray areas at different color temperatures. It can be observed that the gray auxiliary grid is distorted or stretched near the calibration points, indicating that the prediction function has good locality, and mapping at a certain color temperature does not affect the sampling results at other color temperatures. Furthermore, based on the distribution of sampling points after mapping, color temperatures prone to metamerism are identified, and special processing is applied to them, specifically reducing the color compensation weight corresponding to ultra-wide-angle or wide-angle lens modules.

[0194] This scenario-based embodiment proposes a novel color mapping scheme by improving the existing MCSS hard synchronization system. This invention utilizes the collaborative efforts of multiple lens modules to acquire higher-dimensional statistical information, performing color correction on lens modules with small FOVs. This significantly optimizes power consumption increments while resolving color jump issues caused by lens module switching during SAT (Simultaneous Attention Scale) and improving the color accuracy of small FOV lens modules.

[0195] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0196] As shown in Figure 9, the image processing apparatus of this application embodiment may include:

[0197] The first acquisition module 901 is used to acquire first color statistics information of the target lens module of the electronic device at a first moment when the electronic device displays an image captured by the first lens module, wherein the target lens module includes the second lens module.

[0198] The first determining module 902 is configured to determine the first color compensation information mapped by the first color statistics information according to the first color mapping relationship, wherein the first color mapping relationship is the color mapping relationship of the electronic device, and the color mapping relationship is the mapping relationship between the color statistics information of the target lens module and the color compensation information of the first lens module; the first color compensation information is the color compensation information of the first lens module of the electronic device at the first moment.

[0199] The first white balance processing module 903 is used to perform white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the first color compensation information.

[0200] In some embodiments, the image processing apparatus further includes:

[0201] The second determining module is used to determine the color calibration coefficient of the electronic device relative to the reference device using the color statistical information of the electronic device and the color statistical information of the reference device;

[0202] The third determining module is used to determine the first color mapping relationship by using the color calibration coefficient and the reference color mapping relationship, wherein the reference color mapping relationship is the color mapping relationship of the reference device.

[0203] In some embodiments, the image processing apparatus further includes:

[0204] The first construction module is used to construct a first matrix using the color statistics information of the target lens module of the reference device;

[0205] The second construction module is used to construct a second matrix using the color statistics information of the first lens module of the reference device, wherein the dimension of the first matrix is ​​greater than or equal to the dimension of the second matrix;

[0206] The third building module is used to construct the initial color mapping relationship;

[0207] The training module is used to train the initial color mapping relationship based on the first matrix and the second matrix to obtain the reference color mapping relationship.

[0208] In some embodiments, the image processing apparatus further includes:

[0209] The first adjustment module is used to adjust the target lens module to a second operating frequency when the electronic device displays an image captured by the first lens module, wherein the second operating frequency is less than the first operating frequency, and the first operating frequency is the operating frequency of the first lens module.

[0210] The fourth determining module is used to determine the image acquisition time of the first lens module and the image acquisition time of the target lens module using the first operating frequency and the second operating frequency, respectively.

[0211] The first acquisition module 901 is specifically used for at least one of the following:

[0212] If the image acquisition time of the target lens module does not include the first time, the color statistics information of the second image acquired by the target lens module at the second time is determined as the first color statistics information, wherein the second time is the time in the image acquisition time of the target lens module that is before the first time and is closest to the first time.

[0213] If the image acquisition time of the target lens module includes the first time, the color statistics information of the third image acquired by the target lens module at the first time is determined as the first color statistics information.

[0214] In some embodiments, the first acquisition module 901 is specifically used for:

[0215] When the image acquisition time of the target lens module includes the first time, the first color statistics information of the target lens module of the electronic device at the first time is obtained;

[0216] The image processing device further includes:

[0217] The second white balance processing module is used to perform white balance processing on the first image captured by the first lens module of the electronic device at the first moment, using second color compensation information, when the image acquisition time of the target lens module does not include the first moment.

[0218] Wherein, the second color compensation information is the color compensation information of the first lens module at the third moment, and the third moment is the moment in the image acquisition time of the first lens module that is before the first moment.

[0219] In some embodiments, the image processing apparatus further includes:

[0220] The second adjustment module is used to adjust the resolution of the second lens module from a first resolution to a second resolution when the electronic device displays an image captured by the first lens module;

[0221] Wherein, the second resolution is less than the first resolution, and the first resolution is the resolution of the second lens module during the process of the electronic device displaying the image captured by the second lens module.

[0222] In some embodiments, the target lens module further includes a third lens module of the electronic device, wherein the field of view of the third lens module is larger than the field of view of the first lens module.

[0223] In some embodiments, the electronic device further includes a third lens module, the field of view of which is larger than that of the first lens module;

[0224] The image processing device further includes:

[0225] The second acquisition module is used to acquire the angle between the target component of the first hyperplane and the target component of the second hyperplane, wherein the target component includes at least one of a principal component and a secondary component; the first hyperplane is a hyperplane formed by the response curves of each color channel corresponding to the second lens module; the second hyperplane is a hyperplane formed by the response curves of each color channel corresponding to the third lens module.

[0226] The fifth determining module is used to determine the second lens module as the target lens module when the included angle is less than the included angle threshold.

[0227] The sixth determining module is used to determine the second lens module and the third lens module as the target lens module when the included angle is greater than or equal to the included angle threshold.

[0228] In some embodiments, when the target lens module is the second lens module, the image processing apparatus further includes:

[0229] The sixth determining module is used to determine the target compensation weight corresponding to the second lens module by using the reference compensation weight corresponding to the second lens module when the color temperature set includes the ambient color temperature of the electronic device, wherein the target compensation weight is less than the reference compensation weight; the color temperature set includes at least one color temperature, and for each of the at least one color temperature, the statistical information of the second lens module and the first lens module has a metamerism problem at the color temperature;

[0230] The first white balance processing module 903 includes:

[0231] The acquisition unit is used to multiply the first color compensation information and the target compensation weight to obtain the target color compensation information;

[0232] The white balance processing unit is used to perform white balance processing on the first image captured by the first lens module of the electronic device at the first moment based on the target color compensation information.

[0233] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0234] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0235] The image processing apparatus provided in this application embodiment can implement the various processes of the method embodiment, and will not be described again here to avoid repetition.

[0236] Optionally, as shown in FIG10, this application embodiment also provides an electronic device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0237] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0238] Figure 11 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of this application.

[0239] The electronic device 1100 includes, but is not limited to, components such as: radio frequency unit 1101, network module 1102, audio output unit 1103, input unit 1104, sensor 1105, display unit 1106, user input unit 1107, interface unit 1108, memory 1109, and processor 1110.

[0240] Those skilled in the art will understand that the electronic device 1100 may also include a power supply (such as a battery) for powering various components. The power supply may be logically connected to the processor 1110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The electronic device structure shown in Figure 11 does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0241] The processor 1110 is used for:

[0242] When the electronic device displays an image captured by the first lens module, the first color statistics of the target lens module of the electronic device at a first moment are obtained, wherein the target lens module includes the second lens module;

[0243] Based on the first color mapping relationship, the first color compensation information mapped by the first color statistics information is determined, wherein the first color mapping relationship is the color mapping relationship of the electronic device, and the color mapping relationship is the mapping relationship between the color statistics information of the target lens module and the color compensation information of the first lens module; the first color compensation information is the color compensation information of the first lens module of the electronic device at the first moment.

[0244] Based on the first color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

[0245] In some embodiments, processor 1110 is configured to:

[0246] Using the color statistics information of the electronic device and the color statistics information of the reference device, the color calibration coefficient of the electronic device relative to the reference device is determined;

[0247] The first color mapping relationship is determined using the color calibration coefficient and the reference color mapping relationship, wherein the reference color mapping relationship is the color mapping relationship of the reference device.

[0248] In some embodiments, processor 1110 is configured to:

[0249] A first matrix is ​​constructed using the color statistics information of the target lens module of the reference device;

[0250] Using the color statistics information of the first lens module of the reference device, a second matrix is ​​constructed, wherein the dimension of the first matrix is ​​greater than or equal to the dimension of the second matrix;

[0251] Establish the initial color mapping relationship;

[0252] Based on the first matrix and the second matrix, the initial color mapping relationship is trained to obtain the reference color mapping relationship.

[0253] In some embodiments, processor 1110 is configured to:

[0254] When the electronic device displays the image captured by the first lens module, the target lens module is adjusted to a second operating frequency, wherein the second operating frequency is less than the first operating frequency, and the first operating frequency is the operating frequency of the first lens module.

[0255] Using the first operating frequency and the second operating frequency, the image acquisition time of the first lens module and the image acquisition time of the target lens module are determined respectively.

[0256] If the image acquisition time of the target lens module does not include the first time, the color statistics information of the second image acquired by the target lens module at the second time is determined as the first color statistics information, wherein the second time is the time in the image acquisition time of the target lens module that is before the first time and is closest to the first time.

[0257] If the image acquisition time of the target lens module includes the first time, the color statistics information of the third image acquired by the target lens module at the first time is determined as the first color statistics information.

[0258] In some embodiments, processor 1110 is configured to:

[0259] When the image acquisition time of the target lens module includes the first time, the first color statistics information of the target lens module of the electronic device at the first time is obtained;

[0260] If the image acquisition time of the target lens module does not include the first time, the first image acquired by the first lens module of the electronic device at the first time is processed for white balance using the second color compensation information.

[0261] Wherein, the second color compensation information is the color compensation information of the first lens module at the third moment, and the third moment is the moment in the image acquisition time of the first lens module that is before the first moment.

[0262] In some embodiments, processor 1110 is configured to:

[0263] When the electronic device displays an image captured by the first lens module, the resolution of the second lens module is adjusted from the first resolution to the second resolution;

[0264] Wherein, the second resolution is less than the first resolution, and the first resolution is the resolution of the second lens module during the process of the electronic device displaying the image captured by the second lens module.

[0265] In some embodiments, the target lens module further includes a third lens module of the electronic device, wherein the field of view of the third lens module is larger than the field of view of the first lens module.

[0266] In some embodiments, the electronic device further includes a third lens module, the field of view of which is larger than that of the first lens module;

[0267] Processor 1110, used for:

[0268] Obtain the angle between the target components of the first hyperplane and the target components of the second hyperplane, wherein the target components include at least one of the principal components and the secondary components; the first hyperplane is a hyperplane formed by the response curves of each color channel corresponding to the second lens module; the second hyperplane is a hyperplane formed by the response curves of each color channel corresponding to the third lens module.

[0269] If the included angle is less than the included angle threshold, the second lens module is determined as the target lens module;

[0270] If the included angle is greater than or equal to the included angle threshold, the second lens module and the third lens module are identified as the target lens module.

[0271] In some embodiments, when the target lens module is the second lens module, the processor 1110 is configured to:

[0272] When the color temperature set includes the ambient color temperature of the electronic device, the target compensation weight corresponding to the second lens module is determined using the reference compensation weight corresponding to the second lens module, wherein the target compensation weight is less than the reference compensation weight; the color temperature set includes at least one color temperature, and for each of the at least one color temperature, the statistical information of the second lens module and the first lens module has a metamerism problem at the color temperature;

[0273] Multiply the first color compensation information and the target compensation weight to obtain the target color compensation information;

[0274] Based on the target color compensation information, white balance processing is performed on the first image captured by the first lens module of the electronic device at the first moment.

[0275] The electronic device 1100 provided in this application embodiment can implement the various processes of the method embodiment, and will not be described again here to avoid repetition.

[0276] It should be understood that, in this embodiment, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042. The GPU 11041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1106 may include a display panel 11061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1107 includes at least one of a touch panel 11071 and other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0277] The memory 1109 can be used to store software programs and various data. The memory 1109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1109 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0278] Processor 1110 may include one or more processing units; optionally, processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1110.

[0279] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0280] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0281] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0282] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0283] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the image processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0284] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0285] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0286] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method applied to an electronic device, the electronic device comprising a first lens module and a second lens module, wherein, The field of view of the first lens module is less than the field of view of the second lens module; the method comprises: In the case that the electronic device displays the image collected by the first lens module, obtaining first color statistical information of a target lens module of the electronic device at a first time, wherein the target lens module comprises the second lens module; According to a first color mapping relationship, determining first color compensation information mapped by the first color statistical information, wherein the first color mapping relationship is a color mapping relationship of the electronic device, the color mapping relationship is a mapping relationship between color statistical information of the target lens module and color compensation information of the first lens module; the first color compensation information is color compensation information of the first lens module of the electronic device at the first time; Based on the first color compensation information, performing white balance processing on a first image collected by the first lens module of the electronic device at the first time.

2. The method of claim 1, before the determining first color compensation information mapped by the first color statistical information according to a first color mapping relationship, the method further comprises: determining a color calibration coefficient of the electronic device relative to a reference device by using color statistical information of the electronic device and color statistical information of the reference device; determining the first color mapping relationship by using the color calibration coefficient and a reference color mapping relationship, wherein the reference color mapping relationship is a color mapping relationship of the reference device.

3. The method of claim 2, before the determining the first color mapping relationship by using the color calibration coefficient and a reference color mapping relationship, the method further comprises: constructing a first matrix by using color statistical information of the target lens module of the reference device; constructing a second matrix by using color statistical information of the first lens module of the reference device, wherein a dimension of the first matrix is greater than or equal to a dimension of the second matrix; constructing an initial color mapping relationship; training the initial color mapping relationship based on the first matrix and the second matrix to obtain the reference color mapping relationship.

4. The method of claim 1, before the obtaining first color statistical information of a target lens module of the electronic device at a first time, the method further comprises: in the case that the electronic device displays the image collected by the first lens module, adjusting the target lens module to a second working frequency, wherein the second working frequency is less than a first working frequency, and the first working frequency is a working frequency of the first lens module; determining an image collection time of the first lens module and an image collection time of the target lens module respectively by using the first working frequency and the second working frequency; the obtaining first color statistical information of a target lens module of the electronic device at a first time comprises at least one of the following: In a case where the image capturing time of the target lens module does not include the first time, color statistical information of a second image captured by the target lens module at a second time is determined as the first color statistical information, where the second time is the time closest to the first time among the image capturing times of the target lens module. In a case where the image capturing time of the target lens module includes the first time, color statistical information of a third image captured by the target lens module at the first time is determined as the first color statistical information.

5. The method of claim 4, wherein, The obtaining of the first color statistical information of the target lens module of the electronic device at the first time includes: In a case where the image capturing time of the target lens module includes the first time, the first color statistical information of the target lens module of the electronic device at the first time is obtained. After the first working frequency and the second working frequency are used to determine the image capturing time of the first lens module and the image capturing time of the target lens module respectively, the method further includes: In a case where the image capturing time of the target lens module does not include the first time, second color compensation information is used to perform white balance processing on a first image captured by the first lens module of the electronic device at the first time. The second color compensation information is color compensation information of the first lens module at a third time, and the third time is a time before the first time among the image capturing times of the first lens module.

6. The method of claim 1, further comprising: In a case where the electronic device displays the image captured by the first lens module, the resolution of the second lens module is adjusted from a first resolution to a second resolution; The second resolution is less than the first resolution, and the first resolution is the resolution of the second lens module in a process in which the electronic device displays the image captured by the second lens module.

7. The method of claim 1, the target lens module further comprising a third lens module of the electronic device, wherein, The field of view of the third lens module is greater than the field of view of the first lens module.

8. The method of claim 1, wherein, The electronic device further includes a third lens module, and the field of view of the third lens module is greater than the field of view of the first lens module. Before the first color statistical information of the target lens module of the electronic device at the first time is obtained, the method further includes: An angle between a target component of a first hyperplane and a target component of a second hyperplane is obtained, where the target component includes at least one of a principal component and a secondary component; the first hyperplane is a hyperplane formed by color channel response curves corresponding to the second lens module; and the second hyperplane is a hyperplane formed by color channel response curves corresponding to the third lens module. In a case where the angle is less than an angle threshold, the second lens module is determined as the target lens module. In a case where the angle is greater than or equal to the angle threshold, the second lens module and the third lens module are determined as the target lens module.

9. The method of claim 1, wherein, when the target lens module is the second lens module, before the white balance processing of the first image captured by the first lens module of the electronic device at the first time based on the first color compensation information, the method further comprises: determining a target compensation weight corresponding to the second lens module by using a reference compensation weight corresponding to the second lens module, when an ambient color temperature of the electronic device is included in a color temperature set, wherein the target compensation weight is less than the reference compensation weight, and the color temperature set includes at least one color temperature, and for each color temperature in the at least one color temperature, statistical information of the second lens module and the first lens module is metachromatic at the color temperature; and wherein the white balance processing of the first image captured by the first lens module of the electronic device at the first time based on the first color compensation information comprises: multiplying the first color compensation information and the target compensation weight to obtain target color compensation information; and performing white balance processing on the first image captured by the first lens module of the electronic device at the first time based on the target color compensation information. The field of view of the first lens module is less than the field of view of the second lens module. The apparatus comprises: a first obtaining module configured to, when the electronic device displays an image captured by the first lens module, obtain first color statistical information of a target lens module of the electronic device at a first time, wherein the target lens module includes the second lens module; a first determining module configured to determine first color compensation information mapped by the first color statistical information according to a first color mapping relationship, wherein the first color mapping relationship is a color mapping relationship of the electronic device, the color mapping relationship is a mapping relationship between color statistical information of the target lens module and color compensation information of the first lens module, and the first color compensation information is color compensation information of the first lens module of the electronic device at the first time; 10.An image processing apparatus applied to an electronic device, the electronic device comprising a first lens module and a second lens module, wherein, a first white balance processing module configured to perform white balance processing on a first image captured by the first lens module of the electronic device at the first time based on the first color compensation information.

11. An electronic device comprising a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the image processing method according to any one of claims 1 to 9.

12. A readable storage medium, wherein the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the image processing method according to any one of claims 1 to 9.

13. A chip comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, the processor is configured to run programs or instructions to implement the steps of the image processing method according to any one of claims 1 to 9. ​ ​ ​ ​ 14. A computer program product, the program product being stored in a storage medium, the program product being executed by at least one processor to implement the steps of the image processing method of any one of claims 1 to 9.

15. An electronic device configured to perform the steps of the image processing method of any one of claims 1 to 9.

Images