Image processing method based on white balance, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method based on white balance, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: determining a current color temperature value of a current scene shot by an old transmission display lens when the old transmission display lens is switched to a new transmission display lens, and generating a target conversion matrix for performing color alignment from the new transmission display lens to the old transmission display lens under the current color temperature value by using a preset color conversion matrix matched with the current color temperature value according to the current color temperature value, a first color matrix of a first picture shot by the new transmission display lens and a second color matrix of a second picture shot by the old transmission display lens. And converting the first color difference matrix of the old transmitting and displaying lens into a second color difference matrix of a new transmitting and displaying lens by using the target conversion matrix, obtaining a second AWB white point of a second picture based on the second color difference matrix, and performing AWB processing on the picture shot by the new transmitting and displaying lens by using the second AWB white point. The image color consistency before and after lens switching can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, electronic device, and storage medium based on white balance. Background Art

[0002] When switching between different camera lenses on a phone, the color consistency of the images displayed in the camera before and after the switch significantly affects the user experience. A key factor affecting image color is Auto White Balance (AWB). AWB's primary function is to make white objects in a scene appear white. When white objects appear correctly white, other colors appear normal. However, due to potential deviations in AWB calculations, white objects may appear slightly tinted, potentially leaning toward yellow, blue, or other hues.

[0003] When switching lenses, assuming you switch from lens A to lens B, the camera screen displays the footage of lens A before the switch, and the camera screen displays the footage of lens B after the switch. In order to maintain color consistency before and after the switch, the color characteristics of lens B need to be aligned with lens A. In this case, lens A before the switch can be called the old display lens, and lens B after the switch can be called the new display lens. The current method is usually to determine the AWB color deviation characteristics of the old display lens, such as the AWB white point, and then use the AWB white point of the old display lens to calculate the AWB white point of the new display lens, and then use the AWB white point of the new display lens to perform AWB correction on the RAW image shot by the new display lens, thereby improving the color consistency of the picture before and after the switch.

[0004] However, if a color-rich image is used when calculating the AWB white point of the new display lens, the calculated AWB white point of the new display lens will be unreliable due to the different color sensitivities of different lenses, resulting in obvious color differences in the images before and after the lens switch. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method, electronic device, and storage medium based on white balance to improve the color consistency of the image before and after lens switching. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides an image processing method based on white balance, comprising: when switching from an old display lens to a new display lens, determining a current color temperature value of a current scene photographed by the old display lens, determining a first color matrix of a first picture photographed by the new display lens for the current scene, and a second color matrix of a second picture photographed by the old display lens for the current scene; obtaining a preset color conversion matrix corresponding to a target standard light source matching the current color temperature value, the preset color conversion matrix being a conversion matrix for color alignment from the new display lens to the old display lens under the target standard light source; generating a target conversion matrix for color alignment from the new display lens to the old display lens under the current color temperature value based on the color temperature value corresponding to the target standard light source and the preset color conversion matrix; determining a first color difference matrix between the first color matrix and a first AWB white point of the first picture; converting the first color difference matrix into a second color difference matrix corresponding to the new display lens based on the target conversion matrix; obtaining a second AWB white point of the second picture based on the second color difference matrix and the second color matrix; and performing AWB processing on the picture photographed by the new display lens using the second AWB white point.

[0007] Using this method, when switching lenses, a color conversion matrix corresponding to a target standard light source matching the current color temperature value can be obtained, thereby generating a target conversion matrix for color alignment between the new and old display lenses at the current color temperature value. This target conversion matrix can reflect the difference in color sensitivity between the new and old display lenses. A first color difference matrix is ​​then determined between the first color matrix of the first image captured by the old display lens and the first AWB white point of the first image, i.e., the color difference between the first image captured by the old display lens and the first AWB white point is taken into account. Based on the target conversion matrix, the first color difference matrix is ​​then converted into a second color difference matrix corresponding to the new display lens. This ensures that the color difference between the second image captured by the new display lens and the second AWB white point of the new display lens is consistent with the aforementioned color difference of the old display lens. This effectively avoids the color sensitivity difference between the new and old display lenses, making the calculated second AWB white point more accurate. The second AWB white point is then used to perform AWB processing on the image captured by the new display lens, thereby reducing the color difference between the images before and after the lens switch.

[0008] In one possible implementation, determining a first color matrix of a first picture captured by a new display lens for a current scene and a second color matrix of a second picture captured by an old display lens for the current scene include: obtaining a first R / G value and a first B / G value of a first rectangular area in the first picture, the first R / G value being the ratio between an average R value and an average G value of the first rectangular area in the first picture, and the first B / G value being the ratio between an average B value and an average G value of the first rectangular area in the first picture; combining the first R / G value and the first B / G value into a first color matrix; obtaining a second R / G value and a second B / G value of a second rectangular area in the second picture, the second rectangular area and the first rectangular area including images of the same object, the second R / G value being the ratio between an average R value and an average G value of the second rectangular area in the second picture, and the second B / G value being the ratio between an average B value and an average G value of the second rectangular area in the second picture; and combining the second R / G value and the second B / G value into a second color matrix.

[0009] By adopting this method, a first color matrix that can reflect the color of the first picture and a second color matrix that can reflect the color of the second picture can be accurately obtained, which can improve the accuracy of the AWB white point of the new display lens obtained by subsequent calculation.

[0010] In one possible implementation, obtaining a preset color conversion matrix corresponding to a target standard light source that matches a current color temperature value includes: obtaining the color temperature value of each standard light source; if the current color temperature value is greater than the highest color temperature value among the color temperature values ​​of each standard light source, using the standard light source corresponding to the highest color temperature value as the target standard light source; if the current color temperature value is less than the lowest color temperature value among the color temperature values ​​of each standard light source, using the standard light source corresponding to the lowest color temperature value as the target standard light source; if the current color temperature value is between the lowest color temperature value and the highest color temperature value, selecting a first color temperature value and a second color temperature value that are adjacent in magnitude to the current color temperature value from the color temperature values ​​of each standard light source, and using a first standard light source corresponding to the first color temperature value and a second standard light source corresponding to the second color temperature value as the target standard light source; and obtaining a preset color conversion matrix corresponding to the target standard light source.

[0011] In this way, the current color temperature value can be used to determine the light source type of the current scene, thereby accurately determining the target light source type that matches the current color temperature value, and accurately obtaining the preset color conversion matrix between the old display lens and the new display lens suitable for the current scene.

[0012] In one possible implementation, based on the color temperature value corresponding to the target standard light source and the preset color conversion matrix, a target conversion matrix for color alignment from the new display lens to the old display lens under the current color temperature value is generated, including: if the target standard light source includes one standard light source, then the preset color conversion matrix corresponding to the target standard light source is used as the target conversion matrix; if the target standard light source includes a first standard light source corresponding to a first color temperature value and a second standard light source corresponding to a second color temperature value, then the preset color conversion matrix corresponding to the first standard light source and the preset color conversion matrix corresponding to the second standard light source are interpolated using the current color temperature value, the first color temperature value, and the second color temperature value to obtain the target conversion matrix.

[0013] In this way, a target conversion matrix that is more in line with the current color temperature value can be generated based on the preset color matrix, thereby improving the accuracy of subsequent use of the target conversion matrix to align the colors of the new and old display lenses, and improving the color consistency of the images captured by the new and old display lenses.

[0014] In one possible implementation, determining a first color difference matrix between the first color matrix and a first AWB white point of the first picture includes: using a difference between the first color matrix and the first AWB white point as the first color difference matrix;

[0015] Converting the first color difference matrix into a second color difference matrix corresponding to the new display lens based on the target conversion matrix includes: multiplying the first color difference matrix by the target conversion matrix to obtain the second color difference matrix;

[0016] Obtaining a second AWB white point of the second picture based on the second color difference matrix and the second color matrix includes: taking a difference between the second color matrix and the second color difference matrix as the second AWB white point.

[0017] In this way, the target conversion matrix can be used to convert the first color difference matrix into the second color difference matrix, so that the color sensitivity reflected by the second color difference matrix is ​​consistent with that reflected by the second color difference matrix. The difference between the second color matrix and the second color difference matrix is ​​then used as the second AWB white point, which can improve the accuracy of the second AWB white point.

[0018] In one possible implementation, a first picture and a second picture include multiple groups of rectangular areas, each group of rectangular areas includes a first rectangular area in the first picture and a second rectangular area in the second picture, and each group of rectangular areas corresponds to a first color matrix, a second color matrix, and a second color difference matrix. Obtaining a second AWB white point of the second picture based on the second color difference matrix and the second color matrix includes: obtaining a third AWB white point of the second picture based on the second color difference matrix and the second color matrix corresponding to each group of rectangular areas; determining a distance between a position of a point represented by the first color matrix corresponding to each group of rectangular areas and a position of the first AWB white point on a two-dimensional plane of R / B and B / G; determining, for each group of rectangular areas, a weight of the third AWB white point corresponding to the group of rectangular areas based on the distance corresponding to the group of rectangular areas, wherein the distance is positively correlated with the weight of the third AWB white point; and performing a weighted summation of the third AWB white points based on the weight of each third AWB white point to obtain the second AWB white point.

[0019] Using this method, when calculating the second AWB white point of the new display lens, multiple groups of rectangular areas in the first and second images can be used to calculate multiple third AWB white points of the new display lens, and then the multiple third AWB white points are weighted and summed to obtain the second AWB white point. Because when determining the weights of each third AWB white point, the shorter the distance between the position of the point represented by R / B and B / G in the first rectangular area of ​​the old display lens and the position of the first AWB white point, the smaller the error caused by the AWB white point conversion of the new and old display lenses through the target conversion matrix, and therefore the higher the weight of the third AWB white point corresponding to the first rectangular area. Using this principle, the error of the second AWB white point obtained by the final weighted summation is small, which can further improve the accuracy of the calculated second AWB white point, thereby reducing the color difference between the images before and after the lens switch.

[0020] In one possible implementation, for each group of rectangular areas, determining the weight of the third AWB white point corresponding to the group of rectangular areas based on the distance corresponding to the group of rectangular areas includes: for each group of rectangular areas, searching for the weight of the distance mapping corresponding to the group of rectangular areas from a preset mapping relationship between distance and weight, and using the weight as the weight of the third AWB white point corresponding to the group of rectangular areas; or

[0021] Determine the sum of the distances corresponding to each group of rectangular areas to obtain a total distance. For each group of rectangular areas, use the ratio of the first difference to the total distance as the weight of the third AWB white point corresponding to the group of rectangular areas. The first difference is the total distance minus the product of the total number of rectangular areas and the distance corresponding to the group of rectangular areas.

[0022] Using this method, since the shorter the distance between the position of the point represented by R / B and B / G in the first rectangular area of ​​the old display lens and the position of the first AWB white point, the smaller the error caused by the AWB white point conversion of the new and old display lenses through the target conversion matrix, through this method, for each group of rectangular areas, the smaller the distance corresponding to the group of rectangular areas can be made, the higher the weight of the corresponding third AWB white point can be correspondingly obtained, and the weight of the third AWB white point corresponding to each group of rectangular areas can be accurately obtained, thereby improving the accuracy of the finally calculated second AWB white point.

[0023] In a possible implementation, the preset color conversion matrix corresponding to the target standard light source is obtained by the following steps: obtaining a first RAW image obtained by photographing a preset standard color card with an old display lens under the target standard light source; obtaining a second RAW image obtained by photographing a preset standard color card with a new display lens under the target standard light source; calculating the R / G value and the B / G value of each color block included in the first RAW image to obtain a first matrix; calculating the R / G value and the B / G value of each color block included in the second RAW image to obtain a second matrix; subtracting the R / G value of the reference color block in the first RAW image from each R / G value included in the first matrix, and subtracting the R / G value of the reference color block in the first RAW image from each R / G value included in the first matrix, and The B / G value of the reference color block in the first RAW image is subtracted from each B / G value to obtain the color characteristic matrix of the old display lens, where the reference color block is a preset colorless color block in the preset standard color card; the R / G value of the reference color block in the second RAW image is subtracted from each R / G value included in the second matrix, and the B / G value of the reference color block in the second RAW image is subtracted from each B / G value included in the second matrix to obtain the color characteristic matrix of the new display lens; based on the color characteristic matrix of the old display lens and the color characteristic matrix of the new display lens, a color conversion matrix for color alignment of the new display lens to the old display lens is calculated to obtain a preset color conversion matrix.

[0024] By adopting this method, the preset color conversion matrix between the old display lens and the new display lens can be obtained in advance, so that when the lens switch occurs, the preset color matrix can be used in time to process the picture taken by the new display lens to ensure the color consistency of the picture taken by the new display lens and the old display lens.

[0025] In one possible implementation, based on the color characteristic matrix of the old display lens and the color characteristic matrix of the new display lens, a color conversion matrix for color alignment from the new display lens to the old display lens is calculated to obtain a preset color conversion matrix, including:

[0026] The preset color conversion matrix is ​​calculated by the following formula: preset color conversion matrix = inv (transposed matrix of the color characteristic matrix of the new display lens * color characteristic matrix of the new display lens) * transposed matrix of the color characteristic matrix of the new display lens * color characteristic matrix of the old display lens, where inv represents the inverse matrix.

[0027] By adopting this method, the preset color conversion matrix can accurately reflect the color sensitivity difference between the new display lens and the old display lens, so that when the lens is switched, the preset color conversion matrix can be used to eliminate the impact of the color sensitivity difference on the picture.

[0028] In a second aspect, an embodiment of the present application provides an electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method described in the first aspect.

[0029] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method described in the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising a computer program. When the computer program is run on an electronic device, the electronic device executes the method described in the first aspect and the second aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect.

[0032] It is understandable that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to perform the methods provided in this application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0034] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0036] Figure 3a A schematic diagram of a picture captured by an old display lens provided in an embodiment of the present application;

[0037] Figure 3b A schematic diagram of a picture captured by the new display lens provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of a shot switching scenario provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of a 24-color card provided in an embodiment of the present application;

[0040] Figure 6 A flowchart of a method for determining a preset color conversion matrix provided in an embodiment of the present application;

[0041] Figure 7 A flowchart of an image processing method based on white balance provided in an embodiment of the present application;

[0042] Figure 8 A flowchart of another image processing method based on white balance provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0044] The method provided in the embodiments of the present application is applied to electronic devices, which may be mobile phones, tablet computers, televisions, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other electronic devices with multiple lenses.

[0045] In order to better understand the embodiments of the present application, the structure of the electronic device according to the embodiments of the present application is introduced below.

[0046] like Figure 1 As shown, Figure 1 A schematic diagram of an electronic device provided in an embodiment of the present application is provided. Figure 1 The electronic device shown may include a processor 110, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, a button 190, a motor 191, an indicator 192, cameras 1-N 193, a display 194, and a subscriber identification module (SIM) card interface 1-N 195, etc.

[0047] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0048] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor, a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0049] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on instruction operation codes and timing signals to complete the control of instruction fetching and execution.

[0050] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0051] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, etc.

[0052] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0053] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0054] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied in electronic devices. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0055] In the application embodiment, the mobile communication module may also be referred to as a cellular module, and the two may be described interchangeably.

[0056] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, a receiver, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0057] The wireless communication module 160 can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (WiFi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be sent from the processor 110, frequency modulate them, amplify them, and convert them into electromagnetic waves for radiation through the antenna 2. In some embodiments, at least some functional modules of the wireless communication module 160 can be set in the processor 110.

[0058] In some embodiments of the present application, an electronic device can establish a wireless connection with another electronic device via the wireless communication module 160 (such as a Bluetooth module or a WLAN module) and the antenna 2 to enable data transmission between the electronic device and the other electronic device. For example, during a voice call, the electronic device can transmit audio data between the other electronic device via the wireless communication module 160.

[0059] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that the electronic device can communicate with the network and other devices through wireless communication technology.

[0060] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only for illustrative purposes and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0061] The charging management module 140 is configured to receive charging input from a charger. While charging the battery 142 , the charging management module 140 can also provide power to the electronic device through the power management module 141 .

[0062] The electronic device implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0063] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device may include 1 or N display screens 194 , where N is a positive integer greater than 1.

[0064] The electronic device can realize the shooting function through the ISP, camera 193, video codec, GPU, display 194 and application processor.

[0065] The ISP is used to process data fed back by the camera 193. The camera 193 is used to capture still images or videos. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than one.

[0066] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency, the DSP performs a Fourier transform on the frequency energy.

[0067] The electronic device can implement audio functions such as music playback, recording, and voice calls through the audio module 170, speakers, receivers, microphones, headphone jacks, and application processors.

[0068] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device.

[0069] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a screen projection function, a network sharing function, etc.), etc. The data storage area can store data created during the use of the electronic device (such as video data, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0070] Keys 190 include a power button, a volume button, and the like. Keys 190 may be mechanical keys or touch-sensitive keys. The electronic device may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device.

[0071] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or touch vibration feedback. Indicator 192 can be an indicator light that can be used to indicate charging status, power level changes, messages, missed calls, notifications, etc.

[0072] SIM card interface 195 is used to connect a SIM card. A SIM card can be connected to and disconnected from the electronic device by inserting or removing it from the SIM card interface 195. An electronic device may support one or N SIM card interfaces, where N is a positive integer greater than one.

[0073] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software system of the electronic device. Figure 2 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers:

[0074] From top to bottom, they are the application layer, application framework layer, Android runtime (Android Runtime) and system library, and kernel layer.

[0075] The application layer can include a series of application packages. Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, Bluetooth, music, video, and short message.

[0076] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and an algorithm framework, etc.

[0077] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0078] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0079] The view system includes visual controls, such as those for displaying text and images. The view system is used to build the application's display interface. A display interface can be composed of one or more views. For example, the display interface for a text notification icon might include a view for displaying text and a view for displaying an image.

[0080] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0081] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0082] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0083] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0084] The core library consists of two parts: one containing the Java language's callable functions and the other the Android core library. The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0085] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0086] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0087] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0088] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing and layer processing.

[0089] A 2D graphics engine is a drawing engine for 2D drawings.

[0090] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, etc.

[0091] To facilitate understanding, the relevant concepts involved in the embodiments of this application are first introduced.

[0092] RGB: represents the colors of the three channels red (Red), green (Green), and blue (Blue).

[0093] AWB white point: The AWB algorithm determines a point on the two-dimensional plane of R / B and B / G, which is called the AWB white point. Using the AWB white point to perform color correction on the RAW image can make white objects in the scene appear correctly white. For example, the R / B value and B / G value of the AWB white point are 0.5 and 1.2 respectively. Through AWB, the pixels with R / B and B / G values ​​of 0.5 and 1.2 can be corrected to have equal RBG values, so that the pixels with R / B and B / G values ​​of 0.5 and 1.2 appear correctly white. Since the G value does not change, the R value of the entire RAW image is multiplied by 1 / 0.5, and the B value of the entire RAW image is multiplied by 1 / 1.2, thereby achieving color correction of the RAW image using the AWB white point.

[0094] In related technologies, when switching lenses, AWB processing can be performed using the image of a specified rectangular area in the picture taken by the old display lens and the image of the corresponding rectangular area in the picture taken by the new display lens. For example, the old display lens and the new display lens shoot the same scene, and the picture taken by the old display lens is as follows: Figure 3a As shown, the picture taken by the new display lens is as follows Figure 3b The two images have different field of view (FOV), so you can select a rectangular area from the image of the old display lens, for example Figure 3a Then find the corresponding rectangular area containing the same object in the picture sent to the new display lens, for example Figure 3b The rectangular area 302 where the same light is located.

[0095] Then, for the new display lens, the average values ​​N_RECT_R, N_RECT_G and N_RECT_B of the R, G and B values ​​in the rectangular area 302 are calculated respectively, and N_RECT_RG=N_RECT_R / N_RECT_G and N_RECT_BG=N_RECT_B / N_RECT_G are defined.

[0096] For the old display lens, the average values ​​of the R value, the G value, and the B value in the rectangular area 301 are calculated respectively, and O_RECT_RG=O_RECT_R / O_RECT_G is defined, and O_RECT_BG=O_RECT_B / N_RECT_G is defined.

[0097] Then, a two-dimensional color plane of R / G and B / G is established, and (N_RECT_RG, N_RECT_BG) and (O_RECT_RG, O_RECT_BG) are points on the two-dimensional color plane respectively. The white point (O_WP_RG, O_WP_BG) of the old display lens can be determined on the two-dimensional color plane. The white point is used to perform AWB correction on (O_RECT_RG, O_RECT_BG) to obtain O_CORR_RG and O_CORR_BG.

[0098] O_CORR_RG=O_RECT_RG / O_WP_RG;

[0099] O_CORR_BG=O_RECT_BG / O_WP_BG.

[0100] In order to make the colors in the rectangular area 302 in the new display lens the same as those in the rectangular area 301 in the old display lens, the AWB correction results O_CORR_RG and O_CORR_BG of the old display lens can be made the same as the AWB correction results N_CORR_RG and N_CORR_BG of the new display lens. Based on this, the following equation can be obtained:

[0101] N_RECT_RG / N_WP_RG=O_RECT_RG / O_WP_RG;

[0102] N_RECT_BG / N_WP_BG=O_RECT_BG / O_WP_BG.

[0103] The above two equations can be used to solve the white point (N_WP_RG, N_WP_BG) of the new display lens, and then AWB correction can be performed on the image captured by the new display lens according to the white point of the new display lens, thereby improving the color consistency between the new display lens and the old display lens.

[0104] Since the AWB calculation is based on colorless objects, in the above process, if the objects within rectangular area 301 and rectangular area 302 are both colorless objects, the AWB white point of the new display lens is calculated in the above manner, and the AWB white point is used to perform AWB correction on the image captured by the new display lens. This can make the color representation of the colorless objects within rectangular area 301 and rectangular area 302 consistent, and thus make the color characteristics of the images captured by the lenses before and after switching close to the same after AWB correction.

[0105] However, if the objects within rectangular area 301 and rectangular area 302 are both colorful, the AWB white point of the new display lens estimated using the above method is unreliable due to the different color sensitivities of different lenses, resulting in obvious color differences in the images before and after the lens switch.

[0106] In order to solve the above problems, an embodiment of the present application provides an image processing method based on white balance, which is introduced in detail below.

[0107] Taking a mobile phone as an example, a mobile phone usually has multiple lenses, such as a 0.5x focal length lens, a 1x focal length lens, a 2x focal length lens, a 3x focal length lens, etc. When the lens is switched, the scene being photographed usually does not change. Figure 4 As shown, Figure 4 This is an example of a lens switching scenario. Before the switch, the camera is using a 1x focal length lens, while after the switch, the camera is using a 3x focal length lens. The scene before and after the switch is the same: a plant. In this case, the 1x focal length lens is the old display lens, and the 3x focal length lens is the new display lens.

[0108] It is understandable that Figure 4 The lenses shown are only examples, and the embodiments of the present application do not limit the number and type of lenses included in the mobile phone.

[0109] The method provided in the embodiment of the present application can determine a preset color conversion matrix for color alignment from the new display lens to the old display lens based on the color sensitivity of the old display lens and the new display lens. Then, when a lens switch occurs, the preset color conversion matrix is ​​used to align the AWB result of the new display lens with the AWB result of the old display, thereby improving the reliability of the calculated AWB white point of the new display lens and reducing the color difference of the picture before and after the lens switch.

[0110] In order to support switching between different lenses of an electronic device, the embodiment of the present application needs to pre-determine a preset color conversion matrix between every two lenses that may be switched, and in order to adapt to different light sources, it needs to pre-determine a preset color conversion matrix between every two lenses that may be switched under various standard light sources.

[0111] For example, standard illuminants include D75, D65, D50, TL84, A, and H.

[0112] These standard light sources can be set up in a light box, and then each light source can be switched in turn. The old and new transmission lenses can be used to take pictures of the preset standard color card in the light box respectively, so as to obtain the first RAW image taken by the old transmission lens and the second RAW image taken by the new transmission lens, and then the first RAW image and the second RAW image can be used to obtain the preset color conversion matrix between the old and new transmission lenses.

[0113] As an example, the preset standard color card can be a 24-color card, such as Figure 5 As shown, Figure 5 The 24-color card shows the color blocks and their numbers, where color blocks 1 to 18 are color blocks of different colors, and color blocks 19 to 24 are black and white blocks with increasing grayscale.

[0114] The following describes a method for determining a preset color conversion matrix between lens A and lens B, taking the old display lens as lens A, the new display lens as lens B, the standard light source as D50 light source, and the preset standard color card as a 24-color card as an example. This method can be executed by an electronic device that needs to switch lenses later, or it can also be executed by other electronic devices, such as desktop computers, servers, etc. Figure 6 As shown, the method includes:

[0115] S601 : Obtain a first RAW image obtained by photographing a 24-color chart with lens A under a D50 light source, and a second RAW image obtained by photographing 24 color blocks with lens B under a D50 light source.

[0116] S602 : Perform black level correction (BLC) and shading correction on the first RAW image and the second RAW image.

[0117] Among them, shading correction includes color shading correction and luma shading correction.

[0118] S603 : Calculate the R / G value and the B / G value of each color block included in the first RAW image, and calculate the R / G value and the B / G value of each color block included in the second RAW image.

[0119] Among them, for the first RAW image, the R average value, G average value and B average value of each color block included in the first RAW image can be calculated. The R / G value of a color block is the ratio of the R average value to the G average value of the color block, and the B / G value of a color block is the ratio of the B average value to the G average value of the color block.

[0120] The R / G values ​​and B / G values ​​of the 24 color blocks included in the first RAW image can form a 24x2 matrix (first matrix), where 24 corresponds to the 24 color blocks and 2 corresponds to the R / G values ​​and B / G values.

[0121] Similarly, for the second RAW image, the R average value, G average value and B average value of each color block included in the second RAW image can be calculated. The R / G value of a color block is the ratio of the R average value to the G average value of the color block, and the B / G value of a color block is the ratio of the B average value to the G average value of the color block.

[0122] The R / G values ​​and B / G values ​​of the 24 color blocks included in the second RAW image can form a 24x2 matrix (second matrix), where 24 corresponds to the 24 color blocks and 2 corresponds to the R / G values ​​and B / G values.

[0123] S604: Subtract the R / G values ​​of the 21st color block from the R / G values ​​of each color block in the first RAW image, and subtract the B / G values ​​of the 21st color block from the B / G values ​​of each color block in the first RAW image, to obtain a color characteristic matrix for lens A. Subtract the R / G values ​​of the 21st color block from the R / G values ​​of each color block in the second RAW image, and subtract the B / G values ​​of the 21st color block from the B / G values ​​of each color block in the second RAW image, to obtain a color characteristic matrix for lens B.

[0124] Among them, color block 21 is a preset reference color block, and the reference color block can be any non-color color block in the 24-color card. The grayscale of color block 21 is in a relatively moderate position in the grayscale range of color blocks 19 to 24, and the color is neither too white nor too black. Therefore, using color block 21 as a reference color block can make subsequent calculation results more accurate.

[0125] It can be understood that the color characteristic matrix of lens A is a 24x2 matrix, and the R / G value and B / G value of the 21st item in the matrix are both 0. In this embodiment of the present application, the color characteristic matrix of lens A can be named rgbg_ratio_refer_21p_A.

[0126] The color matrix of the B lens is a 24x2 matrix. The R / G value and the B / G value of the 21st item in the matrix are both 0. In the embodiment of the present application, the color characteristic matrix of the B lens can be named rgbg_ratio_refer_21p_B.

[0127] S605 : Based on the color characteristic matrix of lens A and the color characteristic matrix of lens B, calculate a color conversion matrix (CM_A2B_D50) for color alignment of lens B to lens A and a color conversion matrix (CM_B2A_D50) for color alignment of lens A to lens B.

[0128] It is understandable that, since when switching lenses, there may be a situation where the lens A switches to the lens B, or there may be a situation where the lens B switches to the lens A, the color conversion matrix for these two situations can be pre-calculated in the embodiment of the present application.

[0129] Among them, CM_A2B_D50=inv(rgbg_ratio_refer_21p_B'*rgbg_ratio_refer_21p_B)*rgbg_ratio_re fer_21p_B'*rgbg_ratio_refer_21p_A;

[0130] CM_A2B_D50=inv(rgbg_ratio_refer_21p_A'*rgbg_ratio_refer_21p_A)*rgbg_ratio_refer r_21p_A'*rgbg_ratio_refer_21p_B.

[0131] In the above two formulas, inv represents the inverse matrix, rgbg_ratio_refer_21p_B' is the transposed matrix of rgbg_ratio_refer_21p_B, and rgbg_ratio_refer_21p_A' is the transposed matrix of rgbg_ratio_refer_21p_A.

[0132] Taking CM_A2B_D50 as an example, the principle of the color conversion matrix is ​​explained. In CM_A2B_D50, lens A is the old display lens, and lens B is the new display lens. The color characteristic matrix rgbg_ratio_refer_21p_B of the new display lens is simply expressed as the source matrix S, and the color characteristic matrix rgbg_ratio_refer_21p_A of the old display lens is simply expressed as the destination matrix. That is, the purpose of color alignment is to align the source matrix S with the destination matrix T. The color conversion matrix is ​​expressed as CM, and then it can be obtained: SxCM=T;

[0133] Multiply both sides of the equation by the transposed matrix S' of S to obtain: S'xSxCM=S'xT;

[0134] Multiply both sides of the equation by the inverse matrix of S'xS, inv(S'xS), to obtain inv(S'xS)x S'xS xCM=inv(S'xS)x S'xT;

[0135] Wherein, inv(S'xS)x S'xS is the unit matrix, and its value is 1. Therefore, CM=inv(S'xS)x S'xT can be obtained.

[0136] Since S represents rgbg_ratio_refer_21p_B, T represents rgbg_ratio_refer_21p_A, and CM represents CM_A2B_D50, substituting these three into the above formula, we can obtain: CM_A2B_D50=inv(rgbg_ratio_refer_21p_B'*rgbg_ratio_refer_21p_B)*rgbg_ratio_refer_21p_B'*rgbg_ratio_refer_21p_A.

[0137] Using the above method, the color conversion matrix under various standard light sources can be calculated for lens A and lens B. After traversing each standard light source in the light box, the following color conversion matrix can be obtained:

[0138] CM_B2A_D75,CM_A2B_D75;

[0139] CM_B2A_D65,CM_A2B_D65;

[0140] CM_B2A_D50,CM_A2B_D50;

[0141] CM_B2A_TL84,CM_A2B_TL84;

[0142] CM_B2A_A,CM_A2B_A;

[0143] CM_B2A_H, CM_A2B_H.

[0144] After obtaining these color conversion matrices, they can be stored in the mobile phone so that the mobile phone can select a suitable color conversion matrix for AWB processing when switching lenses.

[0145] The following describes the image processing method based on white balance when switching lenses, still taking the case of switching from lens A (old display lens) to lens B (new display lens) as an example. Figure 7 As shown, the method includes:

[0146] S701: When switching from lens A to lens B, determine the current color temperature value of the current scene photographed by lens A.

[0147] The electronic device may use the first picture (RAW image) captured by lens A of the current scene to determine the first AWB white point of lens A, and then calculate the current color temperature value of the current scene based on the first AWB white point.

[0148] Among them, the method of determining the AWB white point based on the RAW image and determining the color temperature value based on the AWB white point can refer to the AWB technology, and the embodiments of the present application are not limited to this.

[0149] S702: Obtain a preset color conversion matrix corresponding to a target standard light source that matches the current color temperature value.

[0150] The electronic device pre-stores the color temperature values ​​of various standard light sources.

[0151] For example, the color temperature corresponding to the D75 light source is 7500Kelvin.

[0152] The color temperature corresponding to the D65 light source is 6500Kelvin;

[0153] The color temperature corresponding to the D50 light source is 5000Kelvin;

[0154] The color temperature corresponding to the TL84 light source is 4100Kelvin;

[0155] The color temperature corresponding to the A (Incandescent) light source is 2856Kelvin;

[0156] The color temperature corresponding to the H (Horizon) light source is 2300Kelvin.

[0157] There are three specific situations for obtaining a target standard light source that matches the current color temperature value:

[0158] Case 1: If the current color temperature is greater than the highest color temperature among the standard light sources, the standard light source corresponding to the highest color temperature is used as the target standard light source. For example, if the current color temperature is 7800Kelvin, the D75 light source can be used as the target standard light source, and the preset color conversion matrix CM_A2B_D75 corresponding to the D75 light source is obtained.

[0159] Case 2: If the current color temperature is lower than the lowest color temperature among the standard light sources, the standard light source corresponding to the lowest color temperature is used as the target standard light source. For example, if the current color temperature is 2000Kelvin, light source H is used as the target standard light source, and the preset color conversion matrix CM_A2B_H corresponding to light source H is obtained.

[0160] Case 3: If the current color temperature value is between the lowest color temperature value and the highest color temperature value, a first color temperature value and a second color temperature value that are adjacent to the current color temperature value are selected from the color temperature values ​​of each standard light source, and the first standard light source corresponding to the first color temperature value and the second standard light source corresponding to the second color temperature value are both used as target standard light sources.

[0161] Among them, the first color temperature value is a color temperature value among the color temperature values ​​of each standard light source that is smaller than the current color temperature value and closest to the current color temperature value; the second color temperature value is a color temperature value among the color temperature values ​​of each standard light source that is larger than the current color temperature value and closest to the current color temperature value.

[0162] For example, if the current color temperature is 6200Kelvin, the first color temperature is 5000Kelvin, the first standard illuminant is D50, and the second color temperature is 6500Kelvin, the second standard illuminant is D65. The preset color conversion matrix CM_A2B_D50 for the D50 illuminant and the preset color conversion matrix CM_A2B_D65 for the D65 illuminant can then be obtained.

[0163] In this way, the current color temperature value can be used to determine the light source type of the current scene, thereby accurately determining the target light source type that matches the current color temperature value, and accurately obtaining the preset color conversion matrix between lens A and lens B suitable for the current scene.

[0164] S703: Based on the color temperature value corresponding to the target standard light source and the preset color conversion matrix, generate a target conversion matrix for color alignment from the new display lens to the old display lens at the current color temperature value.

[0165] Specifically, for Case 1 and Case 2 in S702 , the target standard light source includes one standard light source, and the preset color matrix corresponding to the target standard light source can be used as the target conversion matrix.

[0166] For case 3 in S702, if the target standard light source includes a first standard light source corresponding to the first color temperature value and a second standard light source corresponding to the second color temperature value, the preset color conversion matrix corresponding to the first standard light source and the preset color conversion matrix corresponding to the second standard light source are interpolated using the current color temperature value, the first color temperature value, and the second color temperature value to obtain the target conversion matrix.

[0167] That is, the target conversion matrix = the preset color conversion matrix corresponding to the second standard light source * (current color temperature value - first color temperature value) / (second color temperature value - first color temperature value) + the preset color conversion matrix corresponding to the first standard light source * (second color temperature value - current color temperature value) / (second color temperature value - first color temperature value).

[0168] For example, if the current color temperature is 6200Kelvin, the first color temperature is 5000Kelvin, the first standard illuminant is D50, and the second color temperature is 6500Kelvin, the second standard illuminant is D65. The preset color conversion matrix for D50 is CM_A2B_D50, and the preset color conversion matrix for D65 is CM_A2B_D65.

[0169] Then the target conversion matrix CM_A2B=CM_A2B_D65*(6200-5000) / (6500-5000)+CM_A2B_D50(6500-6200) / (6500-5000).

[0170] S704: Calculate a first color matrix of a first picture captured by lens A of the current scene.

[0171] The first color matrix may specifically be a color matrix of a first rectangular area in lens A, and the first rectangular area is a pre-selected rectangular area.

[0172] The first image is specifically a RAW image. The specific calculation process is to calculate the R average value, G average value, and B average value in a first rectangular area of ​​the first image, and then calculate the first R / G value and the first B / G value of the first rectangular area of ​​the first image.

[0173] The first R / G value is the ratio of the R average value to the G average value of the first rectangular area in the first picture;

[0174] The first B / G value is a ratio between an average B value and an average G value of a first rectangular area in the first picture.

[0175] Then, the first R / G value and the first B / G value are combined into a 1x2 matrix (first color matrix), which is named Rec t_rgbg_ratio_A.

[0176] S705 : Calculate a second color matrix of a second picture captured by lens B for the current scene.

[0177] The second color matrix may specifically be a color matrix of a second rectangular area in the B lens, where the second rectangular area is a rectangular area corresponding to the first rectangular area, and the second rectangular area includes images of the same object as the first rectangular area.

[0178] The electronic device can perform feature extraction on the first rectangular area in the first screen, thereby extracting multiple feature points that can represent the first rectangular area, and then use feature matching to find feature points in the second screen that match the multiple feature points in the first rectangular area, thereby determining the second rectangular area in the second screen using the found feature points.

[0179] The process of calculating the second color matrix is: calculating the R average value, G average value and B average value in the second rectangular area of ​​the second picture, and then calculating the second R / G value and the second B / G value of the second rectangular area of ​​the second picture.

[0180] The second R / G value is the ratio between the R average value and the G average value of the second rectangular area in the second picture;

[0181] The second B / G value is a ratio between the B average value and the G average value of the second rectangular area in the second picture.

[0182] The second R / G value and the second B / G value are then combined into a 1x2 matrix (second color matrix), which is named Rec t_rgbg_ratio_B.

[0183] S706 : Obtain a first AWB white point of the first picture of shot A.

[0184] The first AWB white point is a point on the two-dimensional plane of R / B and B / G, including R / B values ​​and B / G values. The R / B values ​​and B / G values ​​can be combined into a 1x2 matrix named WP_rgbg_ratio_A.

[0185] S707 : Calculate a first color difference matrix between the first color matrix and a first AWB white point of the first picture.

[0186] That is, the first color difference matrix is ​​Rect_rgbg_ratio_A-WP_rgbg_ratio_A.

[0187] S708 : Convert the first color difference matrix into a second color difference matrix for lens B based on the target conversion matrix.

[0188] The second color difference matrix is ​​used to represent the color difference between the second color matrix of the B shot and the second AWB white point of the second picture.

[0189] The second color difference matrix can be expressed as WP_diff_A2B, WP_diff_A2B=WP_diff_A2B=(Rect_rgbg_ratio_A-WP_rgbg_ratio_A)*CM_A2B.

[0190] S709 : Obtain a second AWB white point of the second picture of shot B based on the second color difference matrix and the second color matrix.

[0191] Convert the second AWB white point into a 1x2 matrix and name it Prediction_WP_rgbg_ratio_B.

[0192] Prediction_WP_rgbg_ratio_B=Rect_rgbg_ratio_B-WP_diff_A2B.

[0193] S710 , performing AWB processing on the image captured by lens B using the second AWB white point.

[0194] The AWB-processed image can be obtained by dividing the R value of each pixel of the RAW image taken by the B lens by the R / B value included in the second AWB white point, and dividing the B value of each pixel by the B / G value included in the second AWB white point.

[0195] Using this method, when switching lenses, a color conversion matrix corresponding to a target standard light source matching the current color temperature value can be obtained, thereby generating a target conversion matrix for color alignment between the new and old display lenses at the current color temperature value. This target conversion matrix can reflect the difference in color sensitivity between the new and old display lenses. A first color difference matrix is ​​then determined between the first color matrix of the first image captured by the old display lens and the first AWB white point of the first image, i.e., the color difference between the first image captured by the old display lens and the first AWB white point is taken into account. Based on the target conversion matrix, the first color difference matrix is ​​then converted into a second color difference matrix corresponding to the new display lens. This ensures that the color difference between the second image captured by the new display lens and the second AWB white point of the new display lens is consistent with the aforementioned color difference of the old display lens. This effectively avoids the color sensitivity difference between the new and old display lenses, making the calculated second AWB white point more accurate. The second AWB white point is then used to perform AWB processing on the image captured by the new display lens, thereby reducing the color difference between the images before and after the lens switch.

[0196] It should be noted that in Figure 7In the process shown, the example of a group of rectangular areas included in the first and second screens is taken. In actual implementation, there is also a situation where multiple groups of rectangular areas are included, that is, the first screen includes multiple first rectangular areas, and accordingly, the second screen includes second rectangular areas corresponding to each first rectangular area.

[0197] On this basis, if Figure 8 As shown, the method includes the following steps:

[0198] Among them, S801-S803 are the same as the above-mentioned S701-S703.

[0199] S804: Calculate a first color matrix for each first rectangular area in the first picture captured by lens A.

[0200] S805 : Calculate a second color matrix for each second rectangular area in the second picture captured by lens B.

[0201] S806: Obtain the first AWB white point of the first picture of shot A.

[0202] S807 : Calculate a first color difference matrix between each first color matrix and the first AWB white point.

[0203] S808 : Convert each first color difference matrix into a second color difference matrix for the B lens based on the target conversion matrix.

[0204] That is, each group of rectangular areas corresponds to a first color matrix, a second color matrix, a first color difference matrix, and a second color difference matrix.

[0205] S809 : Obtain first to third AWB white points of the second picture based on the second color difference matrix and the second color matrix corresponding to each group of rectangular areas.

[0206] The method of calculating each third AWB white point in this step is the same as the method of calculating the second AWB white point in S709 , and reference may be made to the relevant description in S709 , which will not be repeated here.

[0207] It can be understood that in this step, a third AWB white point can be calculated for each group of rectangular areas.

[0208] S810: Determine the distance between the position of the point represented by the first color matrix corresponding to each group of rectangular areas on the two-dimensional plane of R / B and B / G and the position of the first AWB white point.

[0209] S811 . For each group of rectangular areas, determine the weight of the third AWB white point corresponding to the group of rectangular areas based on the distance corresponding to the group of rectangular areas.

[0210] Among them, there is a positive correlation between the distance and the weight of the third WAB white point.

[0211] The present invention provides the following two methods for determining weights.

[0212] Method 1: For each group of rectangular areas, the weight of the distance mapping corresponding to the group of rectangular areas is found from the preset mapping relationship between distance and weight, and used as the weight of the third AWB white point corresponding to the group of rectangular areas.

[0213] As an example, the preset mapping relationship is shown in Table 1.

[0214] Table 1

[0215] distance Weight <0.1 1.0 0.1～0.2 0.5 0.2～0.4 0.3 0.4～0.8 0.2 >0.8 0.1

[0216] For example, there are three groups of rectangular areas, and the distances between the R / B and B / G positions in the first color matrix of each group of rectangular areas and the first AWB white point position are 0.05, 0.25, and 0.7, respectively. From Table 1, we can see that the weights of the third AWB white point Prediction_WP_rgbg_ratio_B_1, Prediction_WP_rgbg_ratio_B_2, and Prediction_WP_rgbg_ratio_B_3 of each group of rectangular areas are 1.0, 0.3, and 0.2, respectively.

[0217] Method 2: Determine the sum of the distances corresponding to each group of rectangular areas to obtain a distance sum; for each group of rectangular areas, use the wallpaper of the first difference and the distance sum as the weight of the third AWB white point corresponding to the group of rectangular areas.

[0218] The first difference is the sum of the distances minus the product of the total number of groups of rectangular areas and the distances corresponding to the groups of rectangular areas.

[0219] Assume that there are n groups of rectangular areas, the distances corresponding to the respective groups of rectangular areas are D1 to Dn, and the sum of the distances from D1 to Dn is D.

[0220] Then the weight of the third AWB white point corresponding to the first group of rectangular areas is W1 = (D – n*D1) / D;

[0221] The weight of the third AWB white point corresponding to the second set of rectangular areas is W2 = (D – n*D2) / D;

[0222] The weight of the third AWB white point corresponding to the n-th group of rectangular areas is Wn=(D−n*Dn) / D.

[0223] It should be noted that if it is calculated that the weight corresponding to any group of rectangular areas is less than 0, the weight corresponding to the group of rectangular areas may be reset to 0.

[0224] S812 . Perform weighted summation on the third AWB white points based on the weight of each third AWB white point to obtain a second AWB white point.

[0225] For example, assuming that there are n groups of rectangular areas, the third AWB white points corresponding to each group of rectangular areas are Prediction_WP_rgbg_ratio_B_1, Prediction_WP_rgbg_ratio_B_2, ..., Prediction_WP_rgbg_ratio_B_n, and the weights of the third AWB white points corresponding to each group of rectangular areas are W1, W2, ..., Wn, respectively. Then, the second AWB white point == (Prediction_WP_rgbg_ratio_B_1*W1+Prediction_WP_rgbg_ratio_B_2*W2+ ...+Prediction_WP_rgbg_ratio_B_n*Wn) / (W1+W2+ ...+Wn).

[0226] S813 , performing AWB processing on the image captured by lens B using the second AWB white point.

[0227] Using this method, when calculating the second AWB white point of the new display lens, multiple groups of rectangular areas in the first and second images can be used to calculate multiple third AWB white points of the new display lens, and then the multiple third AWB white points are weighted and summed to obtain the second AWB white point. Because when determining the weights of each third AWB white point, the shorter the distance between the position of the point represented by R / B and B / G in the first rectangular area of ​​the old display lens and the position of the first AWB white point, the smaller the error caused by the AWB white point conversion of the new and old display lenses using the target transformation matrix, and therefore the third AWB white point corresponding to the first rectangular area has a higher weight. Using this principle, the error of the second AWB white point obtained by the final weighted summation is small, which can further improve the accuracy of the calculated second AWB white point, thereby reducing the color difference between the images before and after the lens switch.

[0228] In a specific implementation, the present application further provides a computer storage medium including a computer program, wherein when the computer program is executed on an electronic device, the electronic device executes some or all of the steps in the above embodiment. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0229] In a specific implementation, an embodiment of the present application further provides a computer program product, which includes computer program code. When the computer program code runs on an electronic device, the electronic device executes some or all of the steps in the above method embodiment.

[0230] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0231] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0232] Each embodiment in this specification is described in a related manner. Similar portions between the embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so their description is relatively simple. For related portions, refer to the description of the method embodiments.

[0233] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. An image processing method based on white balance, characterized in that: include: When switching from the old display-feeding lens to the new display-feeding lens, determining a current color temperature value of a current scene captured by the old display-feeding lens, determining a first color matrix of a first image captured by the new display-feeding lens of the current scene, and determining a second color matrix of a second image captured by the old display-feeding lens of the current scene; Obtaining a preset color conversion matrix corresponding to a target standard light source that matches the current color temperature value, wherein the preset color conversion matrix is ​​a conversion matrix for color alignment between the new display lens and the old display lens under the target standard light source; Based on the color temperature value corresponding to the target standard light source and the preset color conversion matrix, a target conversion matrix for color alignment between the new display lens and the old display lens is generated at the current color temperature value; determining a first color difference matrix between the first color matrix and a first AWB white point of the first picture; Converting the first color difference matrix into a second color difference matrix corresponding to the new display lens based on the target conversion matrix; obtaining a second AWB white point of the second picture based on the second color difference matrix and the second color matrix; The second AWB white point is used to perform AWB processing on the image captured by the new display lens.

2. The method according to claim 1, characterized in that The determining of a first color matrix of a first picture captured by the new display lens for the current scene, and a second color matrix of a second picture captured by the old display lens for the current scene, includes: Obtaining a first R / G value and a first B / G value of a first rectangular area in the first picture, where the first R / G value is a ratio of an average R value to an average G value of the first rectangular area in the first picture, and the first B / G value is a ratio of an average B value to an average G value of the first rectangular area in the first picture; combining the first R / G value and the first B / G value into the first color matrix; Obtaining second R / G values ​​and second B / G values ​​of a second rectangular area in the second picture, where the second rectangular area and the first rectangular area include images of the same object, the second R / G value being a ratio of an average R value to an average G value of the second rectangular area in the second picture, and the second B / G value being a ratio of an average B value to an average G value of the second rectangular area in the second picture; The second R / G values ​​and the second B / G values ​​are combined into the second color matrix.

3. The method according to claim 1, characterized in that The obtaining of a preset color conversion matrix corresponding to a target standard light source matching the current color temperature value includes: Get the color temperature value of each standard light source; If the current color temperature value is greater than the highest color temperature value among the color temperature values ​​of the standard light sources, the standard light source corresponding to the highest color temperature value is used as the target standard light source; If the current color temperature value is less than the lowest color temperature value among the color temperature values ​​of the standard light sources, the standard light source corresponding to the lowest color temperature value is used as the target standard light source; If the current color temperature value is between the minimum color temperature value and the maximum color temperature value, selecting a first color temperature value and a second color temperature value that are adjacent to the current color temperature value from the color temperature values ​​of the standard light sources, and using a first standard light source corresponding to the first color temperature value and a second standard light source corresponding to the second color temperature value as the target standard light source; Obtain a preset color conversion matrix corresponding to the target standard light source.

4. The method according to claim 3, characterized in that The step of generating a target conversion matrix for color alignment between the new display lens and the old display lens at the current color temperature value based on the color temperature value corresponding to the target standard light source and the preset color conversion matrix includes: If the target standard light source includes one standard light source, a preset color conversion matrix corresponding to the target standard light source is used as the target conversion matrix; If the target standard light source includes a first standard light source corresponding to the first color temperature value and a second standard light source corresponding to the second color temperature value, an interpolation operation is performed on a preset color conversion matrix corresponding to the first standard light source and a preset color conversion matrix corresponding to the second standard light source using the current color temperature value, the first color temperature value, and the second color temperature value to obtain the target conversion matrix.

5. The method according to claim 1, wherein The determining a first color difference matrix between the first color matrix and a first AWB white point of the first picture includes: taking the difference between the first color matrix and the first AWB white point as the first color difference matrix; The converting the first color difference matrix into a second color difference matrix corresponding to the new display lens based on the target conversion matrix includes: multiplying the first color difference matrix by the target transformation matrix to obtain a second color difference matrix; The obtaining, based on the second color difference matrix and the second color matrix, a second AWB white point of the second picture includes: The difference between the second color matrix and the second color difference matrix is ​​used as the second AWB white point.

6. The method according to claim 2, characterized in that The first picture and the second picture include multiple groups of rectangular areas, each group of rectangular areas includes a first rectangular area in the first picture and a second rectangular area in the second picture, and each group of rectangular areas corresponds to a first color matrix, a second color matrix, and a second color difference matrix; The obtaining, based on the second color difference matrix and the second color matrix, a second AWB white point of the second picture includes: Obtaining a third AWB white point of the second picture based on the second color difference matrix and the second color matrix corresponding to each group of rectangular areas; Determine the distance between the position of the point represented by the first color matrix corresponding to each group of rectangular areas and the position of the first AWB white point on the two-dimensional plane of R / B and B / G; For each group of rectangular areas, determining a weight of a third AWB white point corresponding to the group of rectangular areas based on a distance corresponding to the group of rectangular areas, wherein the distance and the weight of the third AWB white point are positively correlated; Based on the weight of each third AWB white point, weighted summation is performed on the third AWB white points to obtain the second AWB white point.

7. The method according to claim 6, characterized in that The step of determining, for each group of rectangular areas, the weight of the third AWB white point corresponding to the group of rectangular areas based on the distances corresponding to the group of rectangular areas, includes: For each set of rectangular areas, find the weight of the distance mapping corresponding to the set of rectangular areas from the preset mapping relationship between distance and weight, and use it as the weight of the third AWB white point corresponding to the set of rectangular areas; or Determine the sum of the distances corresponding to each group of rectangular areas and obtain the total distance; For each group of rectangular areas, a ratio of the first difference to the total distance is used as the weight of the third AWB white point corresponding to the group of rectangular areas, where the first difference is the total distance minus the product of the total number of rectangular areas and the distance corresponding to the group of rectangular areas.

8. The method according to claim 1, characterized in that The preset color conversion matrix corresponding to the target standard light source is obtained by the following steps: Obtaining a first RAW image obtained by photographing a preset standard color chart under the target standard light source using the old display lens; Obtaining a second RAW image obtained by photographing the preset standard color card with the new display lens under the target standard light source; Calculating R / G values ​​and B / G values ​​of each color block included in the first RAW image to obtain a first matrix; Calculating R / G values ​​and B / G values ​​of each color block included in the second RAW image; Get the second matrix; subtracting the R / G value of a reference color patch in the first RAW image from each R / G value included in the first matrix, and subtracting the B / G value of a reference color patch in the first RAW image from each B / G value included in the first matrix, to obtain a color characteristic matrix of the old display lens, where the reference color patch is a preset colorless color patch in the preset standard color card; subtracting the R / G value of the reference color block in the second RAW image from each R / G value included in the second matrix, and subtracting the B / G value of the reference color block in the second RAW image from each B / G value included in the second matrix, to obtain a color characteristic matrix of the new display lens; Based on the color characteristic matrix of the old display lens and the color characteristic matrix of the new display lens, a color conversion matrix for color alignment of the new display lens to the old display lens is calculated to obtain the preset color conversion matrix.

9. The method according to claim 8, characterized in that The step of calculating a color conversion matrix for color alignment from the new display lens to the old display lens based on the color characteristic matrix of the old display lens and the color characteristic matrix of the new display lens to obtain the preset color conversion matrix includes: The preset color conversion matrix is ​​calculated by the following formula: The preset color conversion matrix = inv (the transposed matrix of the color characteristic matrix of the new display lens * the color characteristic matrix of the new display lens) * the transposed matrix of the color characteristic matrix of the new display lens * the color characteristic matrix of the old display lens, where inv represents obtaining the inverse matrix.

10. An electronic device, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that The invention comprises a computer program, which, when running on an electronic device, causes the electronic device to perform the method according to any one of claims 1 to 9.

Images