A white balance-based image processing method, electronic device, and storage medium

Abstract

This application provides a white balance-based image processing method, electronic device, and storage medium, relating to the field of image processing technology. The method includes: when switching from an old display lens to a new display lens, determining the current color temperature value of the current scene captured by the old display lens, as well as the first color matrix of the first image captured by the new display lens and the second color matrix of the second image captured by the old display lens; using a preset color conversion matrix matched to the current color temperature value, generating a target conversion matrix for color alignment from the new display lens to the old display lens at the current color temperature value; using this target conversion matrix to convert the first color difference matrix of the old display lens to the second color difference matrix of the new display lens; obtaining the second AWB white point of the second image based on the second color difference matrix; and using the second AWB white point to perform AWB processing on the image captured by the new display lens. This can improve the color consistency of the image before and after lens switching.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device and storage medium based on white balance. Background Technology

[0002] When switching between different camera lenses on a mobile phone, the consistency of colors displayed in the camera's image before and after the switch significantly impacts the user experience. One crucial factor affecting image color is Auto White Balance (AWB). AWB's primary function is to ensure that white objects in a scene appear white; only when white objects are correctly displayed can other colors be shown correctly. However, due to potential inaccuracies in AWB calculations, white objects may exhibit a slight color cast, potentially leaning towards yellow, blue, or other colors.

[0003] When switching lenses, assuming a switch from lens A to lens B, the camera screen displays the content captured by lens A before the switch and the content captured by lens B after the switch. To maintain color consistency before and after the switch, the color characteristics of lens B need to be aligned with those of lens A. In this case, lens A before the switch can be called the old display lens, and lens B after the switch can be called the new display lens. Currently, the common approach is to determine the AWB color cast characteristics of the old display lens, such as its AWB white point, and then use this AWB white point to calculate the AWB white point of the new display lens. This AWB white point is then used to perform AWB correction on the RAW image captured by the new display lens, thereby improving color consistency before and after the switch.

[0004] However, if a color-rich image is used when calculating the AWB white point of the new display lens, the different color sensitivities of different lenses will lead to unreliable AWB white point calculations for the new display lens, resulting in significant color differences in the images before and after lens switching. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method, electronic device, and storage medium based on white balance to improve the color consistency of the image before and after lens switching. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a white balance-based image processing method, comprising: when switching from an old display lens to a new display lens, determining the current color temperature value of the current scene captured by the old display lens, determining a first color matrix of a first image of the current scene captured by the new display lens, and a second color matrix of a second image of the current scene captured by the old display lens; obtaining a preset color conversion matrix corresponding to a target standard light source matching the current color temperature value, wherein the preset color conversion matrix is ​​a conversion matrix for color alignment from the new display lens to the old display lens under the target standard light source; generating a target conversion matrix for color alignment from the new display lens to the old display lens under the current color temperature value based on the color temperature value corresponding to the target standard light source and the preset color conversion matrix; determining a first color difference matrix between the first color matrix and a first AWB white point of the first image; converting the first color difference matrix into a second color difference matrix corresponding to the new display lens based on the target conversion matrix; obtaining a second AWB white point of the second image based on the second color difference matrix and the second color matrix; and performing AWB processing on the image captured by the new display lens using the second AWB white point.

[0007] Using this method, when switching lenses, the color conversion matrix corresponding to the target standard light source matching the current color temperature value can be obtained. This generates a target conversion matrix for color alignment from the new display lens to the old display lens at the current color temperature value. This target conversion matrix reflects the difference in color sensitivity between the new and old display lenses. Then, a first color difference matrix is ​​determined between the first color matrix of the first image captured by the old display lens and the first AWB white point of the first image, taking into account the color difference between the first image captured by the old display lens and the first AWB white point. Based on the target conversion matrix, the first color difference matrix is ​​converted into a second color difference matrix corresponding to the new display lens. This ensures that the color difference between the second image captured by the new display lens and the second AWB white point of the new display lens remains consistent with the aforementioned color difference of the old display lens, effectively avoiding the color sensitivity difference between the new and old display lenses. This makes the calculated second AWB white point more accurate, and by using the second AWB white point to perform AWB processing on the image captured by the new display lens, the color difference between the images before and after lens switching can be reduced.

[0008] In one possible implementation, determining a first color matrix for a first image captured by a new display lens of the current scene, and a second color matrix for a second image captured by an old display lens of the current scene, includes: obtaining a first R / G value and a first B / G value for a first rectangular region in the first image, wherein the first R / G value is the ratio between the average R value and the average G value of the first rectangular region in the first image, and the first B / G value is the ratio between the average B value and the average G value of the first rectangular region in the first image; combining the first R / G value and the first B / G value into a first color matrix; obtaining a second R / G value and a second B / G value for a second rectangular region in the second image, wherein the second rectangular region includes images of the same object as the first rectangular region, wherein the second R / G value is the ratio between the average R value and the average G value of the second rectangular region in the second image, and the second B / G value is the ratio between the average B value and the average G value of the second rectangular region in the second image; and combining the second R / G value and the second B / G value into a second color matrix.

[0009] Using this method, a first color matrix that reflects the colors of the first image and a second color matrix that reflects the colors of the second image can be accurately obtained, which can improve the accuracy of the AWB white point of the newly sent display lens obtained in subsequent calculations.

[0010] In one possible implementation, obtaining the preset color conversion matrix corresponding to the target standard light source matching the current color temperature value includes: obtaining the color temperature value of each standard light source; if the current color temperature value is greater than the highest color temperature value among the color temperature values ​​of each standard light source, then the standard light source corresponding to the highest color temperature value is taken as the target standard light source; if the current color temperature value is less than the lowest color temperature value among the color temperature values ​​of each standard light source, then the standard light source corresponding to the lowest color temperature value is taken as the target standard light source; if the current color temperature value is between the lowest and highest color temperature values, then a first color temperature value and a second color temperature value adjacent in magnitude to the current color temperature value are selected from the color temperature values ​​of each standard light source, and both the first standard light source corresponding to the first color temperature value and the second standard light source corresponding to the second color temperature value are taken as the target standard light source; and obtaining the preset color conversion matrix corresponding to the target standard light source.

[0011] In this way, the type of light source in the current scene can be determined by using the current color temperature value, thereby accurately determining the target light source type that matches the current color temperature value, and thus accurately obtaining the preset color conversion matrix between the old and new display lenses suitable for the current scene.

[0012] In one possible implementation, based on the color temperature value corresponding to the target standard light source and a preset color conversion matrix, a target conversion matrix is ​​generated for color alignment from the new display lens to the old display lens at the current color temperature value. This includes: if the target standard light source includes one standard light source, then the preset color conversion matrix corresponding to the target standard light source is used as the target conversion matrix; if the target standard light source includes a first standard light source corresponding to a first color temperature value and a second standard light source corresponding to a second color temperature value, then the current color temperature value, the first color temperature value, and the second color temperature value are used to perform interpolation operations on the preset color conversion matrix corresponding to the first standard light source and the preset color conversion matrix corresponding to the second standard light source to obtain the target conversion matrix.

[0013] In this way, a target conversion matrix that is more in line with the current color temperature value can be generated based on the preset color matrix, thereby improving the accuracy of color alignment between the new and old display lenses using the target conversion matrix and improving the color consistency of the images captured by the new and old display lenses.

[0014] In one possible implementation, determining the first color difference matrix between the first color matrix and the first AWB white point of the first image includes: using the difference between the first color matrix and the first AWB white point as the first color difference matrix.

[0015] The process of converting the first color difference matrix into the second color difference matrix corresponding to the new display lens based on the target transformation matrix includes: multiplying the first color difference matrix with the target transformation matrix to obtain the second color difference matrix;

[0016] Based on the second color difference matrix and the second color matrix, the second AWB white point of the second image is obtained, including: taking the difference between the second color matrix and the second color difference matrix as the second AWB white point.

[0017] Using this method, the first color difference matrix can be converted into a second color difference matrix using the target transformation matrix, so that the color sensitivity reflected by the second color difference matrix is ​​consistent with that of the second color difference matrix. Then, the difference between the second color matrix and the second color difference matrix can be used as the second AWB white point, which can improve the accuracy of the second AWB white point.

[0018] In one possible implementation, the first and second frames include multiple sets of rectangular regions. Each set of rectangular regions includes a first rectangular region in the first frame and a second rectangular region in the second frame. Each set of rectangular regions corresponds to a first color matrix, a second color matrix, and a second color difference matrix. Based on the second color difference matrix and the second color matrix, the second AWB white point of the second frame is obtained, including: obtaining a third AWB white point of the second frame based on the second color difference matrix and the second color matrix corresponding to each set of rectangular regions, respectively; determining the distance between the position of the point represented by the first color matrix corresponding to each set of rectangular regions on the R / B and B / G two-dimensional plane and the position of the first AWB white point; for each set of rectangular regions, determining the weight of the third AWB white point corresponding to the set of rectangular regions based on the distance corresponding to the set of rectangular regions, wherein the distance and the weight of the third AWB white point are positively correlated; and performing a weighted summation of the third AWB white points based on the weight of each third AWB white point to obtain the second AWB white point.

[0019] Using this method, when calculating the second AWB white point of the new display lens, multiple sets of rectangular regions in the first and second frames can be used to calculate multiple third AWB white points of the new display lens. These multiple third AWB white points are then weighted and summed to obtain the second AWB white point. Because the shorter the distance between the position of the point represented by R / B and B / G in the first rectangular region of the old display lens and the position of the first AWB white point when determining the weight of each third AWB white point, the smaller the error caused by the AWB white point conversion between the old and new display lenses through the target transformation matrix. Therefore, the weight of the third AWB white point corresponding to this first rectangular region is higher. Using this principle, the error of the second AWB white point obtained by weighted summation is smaller, which can further improve the accuracy of the calculated second AWB white point, thereby reducing the color difference between the images before and after lens switching.

[0020] In one possible implementation, for each group of rectangular regions, the weight of the third AWB white point corresponding to that group of rectangular regions is determined based on the distances corresponding to those regions. This includes: for each group of rectangular regions, finding the weights of the distance mappings corresponding to that group of rectangular regions from a preset mapping relationship between distance and weights, and using these weights as the weights of the third AWB white point corresponding to that group of rectangular regions; or...

[0021] Determine the sum of distances corresponding to each group of rectangular regions to obtain the total distance; for each group of rectangular regions, use the ratio of the first difference to the total distance as the weight of the third AWB white point corresponding to that group of rectangular regions. The first difference is the sum of distances minus the product of the total number of groups of rectangular regions and the distances corresponding to that group of rectangular regions.

[0022] Using this method, the shorter the distance between the position of the point represented by R / B and B / G in the first rectangular area of ​​the old display lens and the position of the first AWB white point, the smaller the error generated by the conversion of the AWB white point of the new and old display lenses through the target conversion matrix. Therefore, by using this method, for each group of rectangular areas, the smaller the distance corresponding to that group of rectangular areas, the higher the weight of the corresponding third AWB white point. This allows for the accurate determination of the weight of the third AWB white point corresponding to each group of rectangular areas, thereby improving the accuracy of the final calculated second AWB white point.

[0023] In one possible implementation, the preset color conversion matrix corresponding to the target standard light source is obtained through the following steps: acquiring a first RAW image obtained by photographing a preset standard color chart under the target standard light source using an old display lens; acquiring a second RAW image obtained by photographing the preset standard color chart under the target standard light source using a new display lens; calculating the R / G value and B / G value of each color patch included in the first RAW image to obtain a first matrix; calculating the R / G value and B / G value of each color patch included in the second RAW image to obtain a second matrix; subtracting the R / G value of the reference color patch in the first RAW image from each R / G value included in the first matrix, and then converting the first matrix into a second matrix. Subtracting the B / G value of the reference color patch in the first RAW image from each B / G value yields the color characteristic matrix of the old display lens. The reference color patch is a preset colorless patch in the preset standard color chart. Subtracting the R / G value of the reference color patch in the second RAW image from each R / G value in the second matrix, and subtracting the B / G value of the reference color patch in the second RAW image from each B / G value in the second matrix, yields the color characteristic matrix of the new display lens. Based on the color characteristic matrix of the old display lens and the color characteristic matrix of the new display lens, a color conversion matrix for color alignment from the new display lens to the old display lens is calculated, resulting in a preset color conversion matrix.

[0024] Using this method, a preset color conversion matrix between the old and new display lenses can be obtained in advance. This allows the preset color matrix to be used to process the images captured by the new display lens in a timely manner when a lens switch occurs, ensuring color consistency between the images captured by the new and old display lenses.

[0025] In one possible implementation, based on the color characteristic matrix of the old display lens and the color characteristic matrix of the new display lens, a color conversion matrix for color alignment from the new display lens to the old display lens is calculated, resulting in a preset color conversion matrix, including:

[0026] The preset color conversion matrix is ​​calculated using the following formula: Preset color conversion matrix = inv(transpose of the color characteristic matrix of the new display lens * color characteristic matrix of the new display lens) * transpose of the color characteristic matrix of the new display lens * color characteristic matrix of the old display lens, where inv represents taking the inverse matrix.

[0027] This method allows the preset color conversion matrix to accurately reflect the color sensitivity difference between the new and old display lenses, thereby eliminating the impact of color sensitivity differences on the image when switching lenses.

[0028] In a second aspect, embodiments of this application provide an electronic device, including: one or more processors and a memory; the memory is coupled to one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and one or more processors call the computer instructions to cause the electronic device to perform the method described in the first aspect.

[0029] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect.

[0030] Fourthly, embodiments of this application provide a computer-readable storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform the methods described in the first and second aspects.

[0031] Fifthly, embodiments of this application provide a computer program product, which includes computer program code that, when executed on an electronic device, causes the electronic device to perform the method described in the first aspect.

[0032] Understandably, the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0034] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0035] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0036] Figure 3a A schematic diagram of an image captured by a conventional display lens as provided in an embodiment of this application;

[0037] Figure 3b A schematic diagram of an image captured by the new display lens provided in an embodiment of this application;

[0038] Figure 4 A schematic diagram of a camera switching scene provided in an embodiment of this application;

[0039] Figure 5 A schematic diagram of a 24-color card provided for an embodiment of this application;

[0040] Figure 6 A flowchart illustrating a method for determining a preset color conversion matrix provided in this application embodiment;

[0041] Figure 7 A flowchart illustrating a white balance-based image processing method provided in this application embodiment;

[0042] Figure 8 A flowchart of another white balance-based image processing method provided in an embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0044] The method provided in this application embodiment is applied to electronic devices, which can be mobile phones, tablets, televisions, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other electronic devices with multiple lenses.

[0045] To better understand the embodiments of this application, the structure of the electronic device of the embodiments of this application is described below.

[0046] like Figure 1 As shown, Figure 1 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 1 The electronic device shown may include a processor 110, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, sensor module 180, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, button 190, motor 191, indicator 192, camera 1-N 193, display screen 194, and subscriber identification module (SIM) card interface 1-N 195, etc.

[0047] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0048] Processor 110 may include one or more processing units, such as application processors, modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, memory, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0049] The controller can serve as the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0050] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0051] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include inter-integrated circuit (I2C) interfaces, inter-integrated circuit sound (I2S) interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, etc.

[0052] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0053] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0054] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G in electronic devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0055] In the embodiments of the application, the mobile communication module may also be referred to as a cellular module, and the two can be described interchangeably.

[0056] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to a speaker, receiver, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0057] The wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (WiFi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2. In some embodiments, at least some functional modules of the wireless communication module 160 can be housed within processor 110.

[0058] In some embodiments of this application, an electronic device can establish a wireless connection with other electronic devices via a wireless communication module 160 (such as a Bluetooth module and a WLAN module) and an antenna 2 to enable data transmission between the electronic device and other electronic devices. For example, during a voice call, the electronic device can transmit audio data with other electronic devices via the wireless communication module 160.

[0059] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the electronic device to communicate with networks and other devices via wireless communication technology.

[0060] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0061] The charging management module 140 receives charging input from the charger. While charging the battery 142, the charging management module 140 can also supply power to electronic devices through the power management module 141.

[0062] Electronic devices implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0063] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device may include one or N display screens 194, where N is a positive integer greater than 1.

[0064] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0065] The ISP is used to process data fed back by the camera 193. The camera 193 is used to capture still images or videos. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0066] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.

[0067] Electronic devices can implement audio functions through audio modules 170, speakers, receivers, microphones, headphone jacks, and application processors. Examples include music playback, recording, and voice calls.

[0068] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the electronic device.

[0069] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as screen mirroring, network sharing, etc.), etc. The data storage area may store data created during the use of the electronic device (such as video data, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0070] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. The electronic device can receive button input and generate key signal inputs related to user settings and function control of the electronic device.

[0071] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, or to indicate messages, missed calls, notifications, etc.

[0072] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the electronic device. The electronic device can support one or N SIM card interfaces, where N is a positive integer greater than 1.

[0073] The software system of the aforementioned electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to illustrate the software system of the electronic device. Figure 2 As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers.

[0074] From top to bottom, they are the application layer, application framework layer, Android Runtime and system libraries, and kernel layer.

[0075] The application layer can include a series of application packages. For example... Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, Bluetooth, music, video, and SMS.

[0076] The application framework layer provides an Application Programming Interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, and algorithm framework, etc.

[0077] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0078] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0079] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build the display interface of an application. The display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0080] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0081] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and so on.

[0082] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0083] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0084] The core library consists of two parts: one part contains the functionalities that the Java language needs to call, and the other part is the Android core library. The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0085] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0086] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0087] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0088] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0089] A 2D graphics engine is a drawing engine for 2D drawing.

[0090] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0091] To facilitate understanding, the relevant concepts involved in the embodiments of this application will be introduced first.

[0092] RGB: Represents the colors of the three channels: Red, Green, and Blue.

[0093] AWB White Point: The AWB algorithm determines a point on the 2D plane of R / B and B / G values, called the AWB white point. Using the AWB white point to correct the color of a RAW image ensures that white objects in a scene appear correctly as white. For example, if the R / B and B / G values ​​of the AWB white point are 0.5 and 1.2 respectively, AWB can correct pixels with R / B and B / G values ​​of 0.5 and 1.2 to have equal R, B, and G values, thus ensuring that these pixels correctly appear white. Since the G value remains unchanged, the R value of the entire RAW image is multiplied by 1 / 0.5, and the B value is multiplied by 1 / 1.2, thereby achieving color correction of the RAW image using the AWB white point.

[0094] In related technologies, during lens switching, AWB processing can be performed using the image of a specified rectangular area within the frame captured by the old display lens and the corresponding rectangular area within the frame captured by the new display lens. For example, if the old and new display lenses capture the same scene, the image captured by the old display lens might be as follows: Figure 3a As shown, the image captured by the new display lens is as follows: Figure 3b As shown. The two images have different fields of view (FOV). A rectangular area can be selected from the image of the old display lens, for example... Figure 3a In the rectangular area 301 where one of the lights is located, the corresponding rectangular area containing the same object is found in the image of the newly sent display lens, for example... Figure 3b The rectangular area 302 where the same light is located.

[0095] Then, for the new display lens, calculate the average values ​​of R, G and B within the rectangular region 302, namely N_RECT_R, N_RECT_G and N_RECT_B, respectively. Define N_RECT_RG = N_RECT_R / N_RECT_G and N_RECT_BG = N_RECT_B / N_RECT_G.

[0096] For the old display lens, calculate the average value of R, G and B values ​​within the rectangular area 301, and define O_RECT_RG = O_RECT_R / O_RECT_G and O_RECT_BG = O_RECT_B / N_RECT_G.

[0097] Then, a two-dimensional color plane for R / G and B / G is established. (N_RECT_RG, N_RECT_BG) and (O_RECT_RG, O_RECT_BG) are points on this two-dimensional color plane. The white point (O_WP_RG, O_WP_BG) of the old display lens can be determined on this two-dimensional color plane. Using this white point, AWB correction is performed on (O_RECT_RG, O_RECT_BG) to obtain O_CORR_RG and O_CORR_BG.

[0098] O_CORR_RG=O_RECT_RG / O_WP_RG;

[0099] O_CORR_BG=O_RECT_BG / O_WP_BG.

[0100] In order to make the colors in rectangular region 302 of the new display lens the same as those in rectangular region 301 of the old display lens, the AWB correction results O_CORR_RG and O_CORR_BG of the old display lens can be made the same as the AWB correction results N_CORR_RG and N_CORR_BG of the new display lens. Based on this, the following equation can be obtained:

[0101] N_RECT_RG / N_WP_RG=O_RECT_RG / O_WP_RG;

[0102] N_RECT_BG / N_WP_BG=O_RECT_BG / O_WP_BG.

[0103] The white point (N_WP_RG, N_WP_BG) of the new display lens can be obtained using the above two equations. Then, AWB correction can be performed on the image captured by the new display lens according to the white point of the new display lens, thereby improving the color consistency between the new display lens and the old display lens.

[0104] Since AWB calculation is based on colorless objects, in the above process, if the objects in rectangular areas 301 and 302 are both colorless objects, the AWB white point of the new display lens is calculated in the above manner, and the AWB white point is used to perform AWB correction on the image captured by the new display lens. This makes the color performance of colorless objects in rectangular areas 301 and 302 consistent, and thus makes the color characteristics of the images captured by the lenses before and after switching approximately the same after AWB correction.

[0105] However, if the objects in rectangular areas 301 and 302 are all colorful, the AWB white point of the new display lens estimated by the above method is unreliable due to the different color sensitivities of different lenses, resulting in obvious color differences in the images before and after lens switching.

[0106] To address the aforementioned issues, this application provides an image processing method based on white balance, which will be described in detail below.

[0107] Taking mobile phones as an example, mobile phones typically have multiple lenses, such as 0.5x focal length lenses, 1x focal length lenses, 2x focal length lenses, and 3x focal length lenses. When switching lenses, the scene being photographed usually remains unchanged. Figure 4 As shown, Figure 4 Here's an example of a lens switching scenario: before the switch, the camera uses a 1x focal length lens, and after the switch, it uses a 3x focal length lens. The shooting scene is the same before and after the switch—a plant. In this case, the 1x focal length lens is the old display lens, and the 3x focal length lens is the new display lens.

[0108] Understandable Figure 4 The lens shown is merely an example, and the embodiments of this application do not limit the number and type of lenses included in the mobile phone.

[0109] The method provided in this application can determine a preset color conversion matrix for color alignment from the new display lens to the old display lens based on the color sensitivity of the old and new display lenses. Then, when a lens switch occurs, the preset color conversion matrix is ​​used to align the AWB result of the new display lens to the AWB result of the old display lens, thereby improving the reliability of the calculated AWB white point of the new display lens and reducing the color difference of the image before and after the lens switch.

[0110] In order to support the switching between different lenses of electronic devices, the embodiments of this application need to predetermine the preset color conversion matrix between every two lenses that may switch, and in order to adapt to different light sources, the preset color conversion matrix of every two lenses that may switch under each standard light source needs to be predetermined.

[0111] For example, standard light sources include D75, D65, D50, TL84, A, and H light sources.

[0112] These standard light sources can be set inside the lightbox, and then each light source can be switched in turn. The old and new display lenses can be used to take pictures of the preset standard color card in the lightbox, thereby obtaining the first RAW image obtained by the old display lens and the second RAW image obtained by the new display lens. Then, the preset color conversion matrix between the old and new display lenses can be obtained using the first and second RAW images.

[0113] As an example, the default standard color chart can be a 24-color chart, such as... Figure 5 As shown, Figure 5 The diagram shows the color blocks included in the 24-color chart and their numbers. Color blocks 1 to 18 are colored color blocks of different colors, and color blocks 19 to 24 are black and white blocks with increasing gray levels.

[0114] The following example uses the old display lens as lens A, the new display lens as lens B, the standard light source as a D50 light source, and the preset standard color chart as a 24-color chart to illustrate the method for determining the preset color conversion matrix between lens A and lens B. This method can be executed by the electronic device that needs to switch lenses later, or it can be executed by other electronic devices, such as desktop computers, servers, etc. Figure 6 As shown, the method includes:

[0115] S601. Obtain the first RAW image obtained by taking a picture of the 24-color card with lens A under D50 light source, and the second RAW image obtained by taking a picture of the 24-color block with lens B under D50 light source.

[0116] S602. Perform black level correction (BLC) and shading correction on the first RAW image and the second RAW image.

[0117] Shading correction includes color shading correction and luma shading correction.

[0118] S603. Calculate the R / G value and B / G value of each color block included in the first RAW image, and calculate the R / G value and B / G value of each color block included in the second RAW image.

[0119] Specifically, for the first RAW image, the average R, average G, and average B of each color patch included in the first RAW image can be calculated. The R / G value of a color patch is the ratio of the average R to the average G of that color patch, and the B / G value of a color patch is the ratio of the average B to the average G of that color patch.

[0120] The R / G and B / G values ​​of the 24 color patches in the first RAW image can form a 24x2 matrix (the first matrix), where 24 corresponds to the 24 color patches and 2 corresponds to the R / G and B / G values.

[0121] Similarly, for the second RAW image, the average R, average G, and average B of each color block included in the second RAW image can be calculated. The R / G value of a color block is the ratio of the average R to the average G of that color block, and the B / G value of a color block is the ratio of the average B to the average G of that color block.

[0122] The R / G and B / G values ​​of the 24 color patches in the second RAW image can form a 24x2 matrix (the second matrix), where 24 corresponds to the 24 color patches and 2 corresponds to the R / G and B / G values.

[0123] S604. Subtract the R / G value of 21 color patches from the R / G value of each color patch included in the first RAW image, and subtract the B / G value of 21 color patches from the B / G value of each color patch included in the first RAW image to obtain the color characteristic matrix of lens A; subtract the R / G value of 21 color patches from the R / G value of each color patch included in the second RAW image, and subtract the B / G value of 21 color patches from the B / G value of each color patch included in the second RAW image to obtain the color characteristic matrix of lens B.

[0124] Among them, color block 21 is a preset reference color block. The reference color block can be any non-color color block in the 24 color chart. The gray level of color block 21 is in a relatively moderate position in the gray level range of color blocks 19 to 24. The color is neither too white nor too black. Therefore, using color block 21 as a reference color block can make the subsequent calculation results more accurate.

[0125] It is understood that the color characteristic matrix of lens A is a 24x2 matrix, in which the R / G value and B / G value of the 21st item are both 0. In this embodiment, the color characteristic matrix of lens A can be named rgbg_ratio_refer_21p_A.

[0126] The color matrix of the B lens is a 24x2 matrix. The R / G value and B / G value of the 21st item in this matrix are both 0. In this embodiment, the color characteristic matrix of the B lens can be named rgbg_ratio_refer_21p_B.

[0127] S605. Based on the color characteristic matrix of lens A and the color characteristic matrix of lens B, calculate the color transformation matrix (CM_A2B_D50) for color alignment from lens B to lens A and the color transformation matrix (CM_B2A_D50) for color alignment from lens A to lens B.

[0128] It is understandable that, since there may be a situation where the camera switches from lens A to lens B, or from lens B to lens A, the color conversion matrix in this application embodiment can be pre-calculated for both situations.

[0129] Among them, CM_A2B_D50=inv(rgbg_ratio_refer_21p_B'*rgbg_ratio_refer_21p_B)*rgbg_ratio_re fer_21p_B'*rgbg_ratio_refer_21p_A;

[0130] CM_A2B_D50=inv(rgbg_ratio_refer_21p_A'*rgbg_ratio_refer_21p_A)*rgbg_ratio_refer r_21p_A'*rgbg_ratio_refer_21p_B.

[0131] In the two formulas above, inv represents finding the inverse matrix, rgbg_ratio_refer_21p_B' is the transpose of rgbg_ratio_refer_21p_B, and rgbg_ratio_refer_21p_A' is the transpose of rgbg_ratio_refer_21p_A.

[0132] Taking CM_A2B_D50 as an example, the principle of this color conversion matrix is ​​explained. In CM_A2B_D50, lens A is the old display lens and lens B is the new display lens. The color characteristic matrix rgbg_ratio_refer_21p_B of the new display lens can be simply represented as the source matrix S, and the color characteristic matrix rgbg_ratio_refer_21p_A of the old display lens can be simply represented as the destination matrix. That is, the purpose of color alignment is to align the source matrix S with the destination matrix T. If the color conversion matrix is ​​represented as CM, then we can get: SxCM=T;

[0133] Multiplying both sides of the equation by the transpose of S, S', we get: S'xSxCM=S'xT;

[0134] Multiplying both sides of the equation by the inverse matrix of S'xS, inv(S'xS), we get inv(S'xS)x S'xS xCM=inv(S'xS)x S'xT;

[0135] Where inv(S'xS)xS'xS is the identity matrix with a value of 1, so we can get CM=inv(S'xS)xS'xT.

[0136] Since S represents rgbg_ratio_refer_21p_B, T represents rgbg_ratio_refer_21p_A, and CM represents CM_A2B_D50, substituting these three into the above formula, we can obtain: CM_A2B_D50=inv(rgbg_ratio_refer_21p_B'*rgbg_ratio_refer_21p_B)*rgbg_ratio_refer_21p_B'*rgbg_ratio_refer_21p_A.

[0137] Using the above method, color conversion matrices under various standard light sources can be calculated for lenses A and B. After traversing all standard light sources within the lightbox, the following color conversion matrices can be obtained:

[0138] CM_B2A_D75, CM_A2B_D75;

[0139] CM_B2A_D65, CM_A2B_D65;

[0140] CM_B2A_D50, CM_A2B_D50;

[0141] CM_B2A_TL84, CM_A2B_TL84;

[0142] CM_B2A_A, CM_A2B_A;

[0143] CM_B2A_H, CM_A2B_H.

[0144] After obtaining these color conversion matrices, they can be stored in the phone so that the phone can select the appropriate color conversion matrix for AWB processing when switching lenses.

[0145] The following describes the image processing method based on white balance during lens switching, using the example of switching from lens A (old display lens) to lens B (new display lens). Figure 7 As shown, the method includes:

[0146] S701. When switching from lens A to lens B, determine the current color temperature value of the current scene captured by lens A.

[0147] The electronic device can use the first image (RAW image) of the current scene captured by the A lens to determine the first AWB white point of the A lens, and then calculate the current color temperature value of the current scene based on the first AWB white point.

[0148] The methods for determining the AWB white point based on the RAW image and the method for determining the color temperature value based on the AWB white point can refer to the AWB technology, and this application embodiment does not limit them.

[0149] S702. Obtain the preset color conversion matrix corresponding to the target standard light source that matches the current color temperature value.

[0150] The electronic device pre-stores the color temperature values ​​of each standard light source.

[0151] For example, the color temperature of the D75 light source is 7500 Kelvin.

[0152] The color temperature of the D65 light source is 6500 Kelvin.

[0153] The color temperature of the D50 light source is 5000 Kelvin.

[0154] The color temperature of the TL84 light source is 4100 Kelvin.

[0155] The color temperature of the A (Incandescent) light source is 2856 Kelvin;

[0156] The color temperature of the H (Horizon) light source is 2300 Kelvin.

[0157] There are three specific scenarios for obtaining a target standard light source that matches the current color temperature value:

[0158] Case 1: If the current color temperature value is greater than the highest color temperature value among all standard light sources, then the standard light source corresponding to the highest color temperature value will be used as the target standard light source. For example, if the current color temperature value is 7800 Kelvin, then the D75 light source can be used as the target standard light source, and the preset color conversion matrix CM_A2B_D75 corresponding to the D75 light source can be obtained.

[0159] Scenario 2: If the current color temperature value is less than the lowest color temperature value among all standard light sources, then the standard light source corresponding to the lowest color temperature value is used as the target standard light source. For example, if the current color temperature value is 2000 Kelvin, then light source H can be used as the target standard light source, and the preset color conversion matrix CM_A2B_H corresponding to light source H can be obtained.

[0160] Case 3: If the current color temperature value is between the lowest and highest color temperature values, then select the first and second color temperature values ​​that are adjacent to the current color temperature value from the color temperature values ​​of each standard light source, and use the first standard light source corresponding to the first color temperature value and the second standard light source corresponding to the second color temperature value as the target standard light source.

[0161] The first color temperature value is the color temperature value among the standard light sources that is less than the current color temperature value and closest to the current color temperature value; the second color temperature value is the color temperature value among the standard light sources that is greater than the current color temperature value and closest to the current color temperature value.

[0162] For example, if the current color temperature is 6200 Kelvin, then the first color temperature is 5000 Kelvin, the first standard light source is a D50 light source, the second color temperature is 6500 Kelvin, and the second standard light source is a D65 light source. Therefore, the preset color conversion matrix CM_A2B_D50 corresponding to the D50 light source and the preset color conversion matrix CM_A2B_D65 corresponding to the D65 light source can be obtained.

[0163] In this way, the type of light source in the current scene can be determined by using the current color temperature value, thereby accurately determining the target light source type that matches the current color temperature value, so as to accurately obtain the preset color conversion matrix between lens A and lens B suitable for the current scene.

[0164] S703. Based on the color temperature value corresponding to the target standard light source and the preset color conversion matrix, generate a target conversion matrix for color alignment from the new display lens to the old display lens under the current color temperature value.

[0165] Specifically, for cases 1 and 2 in S702, if the target standard light source includes one standard light source, then the preset color matrix corresponding to the target standard light source can be used as the target transformation matrix.

[0166] Regarding case 3 in S702, if the target standard light source includes a first standard light source corresponding to a first color temperature value and a second standard light source corresponding to a second color temperature value, then the current color temperature value, the first color temperature value, and the second color temperature value are used to perform interpolation operations on the preset color transformation matrix corresponding to the first standard light source and the preset color transformation matrix corresponding to the second standard light source to obtain the target transformation matrix.

[0167] That is, the target conversion matrix = the preset color conversion matrix corresponding to the second standard light source * (current color temperature value - first color temperature value) / (second color temperature value - first color temperature value) + the preset color conversion matrix corresponding to the first standard light source * (second color temperature value - current color temperature value) / (second color temperature value - first color temperature value).

[0168] For example, if the current color temperature is 6200 Kelvin, then the first color temperature is 5000 Kelvin, the first standard light source is a D50 light source, the second color temperature is 6500 Kelvin, and the second standard light source is a D65 light source. The preset color conversion matrix corresponding to the D50 light source is CM_A2B_D50, and the preset color conversion matrix corresponding to the D65 light source is CM_A2B_D65.

[0169] Then the target transformation matrix CM_A2B = CM_A2B_D65*(6200-5000) / (6500-5000) + CM_A2B_D50(6500-6200) / (6500-5000).

[0170] S704. Calculate the first color matrix of the first frame captured by lens A of the current scene.

[0171] Specifically, the first color matrix can be the color matrix of the first rectangular area in lens A, and the first rectangular area is a pre-selected rectangular area.

[0172] The first frame is a RAW image. The specific calculation process is as follows: calculate the average R, average G, and average B values ​​within the first rectangular area of ​​the first frame, and then calculate the first R / G value and the first B / G value of the first rectangular area of ​​the first frame.

[0173] Wherein, the first R / G value is the ratio between the average R value and the average G value of the first rectangular region in the first image;

[0174] The first B / G value is the ratio between the average B value and the average G value of the first rectangular area in the first frame.

[0175] Then, the first R / G value and the first B / G value are combined into a 1x2 matrix (the first color matrix), named Record_rgbg_ratio_A.

[0176] S705, Calculate the second color matrix of the second image captured by lens B of the current scene.

[0177] Specifically, the second color matrix can be the color matrix of the second rectangular region in the B lens. The second rectangular region is a rectangular region corresponding to the first rectangular region, and the second rectangular region contains images of the same objects as the first rectangular region.

[0178] The electronic device can perform feature extraction on a first rectangular region in a first screen to extract multiple feature points that can represent the first rectangular region. Then, it can use feature matching to find feature points in a second screen that match the multiple feature points in the first rectangular region, thereby using the found feature points to determine the second rectangular region in the second screen.

[0179] The process of calculating the second color matrix is ​​as follows: calculate the average R, average G, and average B values ​​within the second rectangular area of ​​the second image, and then calculate the second R / G value and the second B / G value of the second rectangular area of ​​the second image.

[0180] Among them, the second R / G value is the ratio between the average R value and the average G value of the second rectangular region in the second image;

[0181] The second B / G value is the ratio between the average B value and the average G value of the second rectangular area in the second image.

[0182] Then, the second R / G value and the second B / G value are combined into a 1x2 matrix (the second color matrix), named Rect_rgbg_ratio_B.

[0183] S706, Obtain the first AWB white point of the first frame of lens A.

[0184] The first AWB white point is a point on the two-dimensional plane of R / B and B / G, including R / B value and B / G value. The R / B value and B / G value can be combined into a 1x2 matrix, named WP_rgbg_ratio_A.

[0185] S707. Calculate the first color difference matrix between the first color matrix and the first AWB white point of the first image.

[0186] That is, the first color difference matrix is ​​Rect_rgbg_ratio_A-WP_rgbg_ratio_A.

[0187] S708, Based on the target transformation matrix, convert the first color difference matrix into the second color difference matrix of the B lens.

[0188] The second color difference matrix is ​​used to represent the color difference between the second color matrix of the B lens and the second AWB white point of the second image.

[0189] The second color difference matrix can be represented as WP_diff_A2B, where WP_diff_A2B = WP_diff_A2B = (Rect_rgbg_ratio_A - WP_rgbg_ratio_A) * CM_A2B.

[0190] S709. Based on the second color difference matrix and the second color matrix, the second AWB white point of the second image of the B lens is obtained.

[0191] The second AWB white point is converted into a 1x2 matrix and named Prediction_WP_rgbg_ratio_B.

[0192] Prediction_WP_rgbg_ratio_B=Rect_rgbg_ratio_B-WP_diff_A2B.

[0193] S710: AWB processing is performed on the image captured by lens B using the second AWB white point.

[0194] Specifically, the R value of each pixel in the RAW image captured by the B lens can be divided by the R / B value included in the second AWB white point, and the B value of each pixel can be divided by the B / G value included in the second AWB white point to obtain the AWB processed image.

[0195] Using this method, when switching lenses, the color conversion matrix corresponding to the target standard light source matching the current color temperature value can be obtained. This generates a target conversion matrix for color alignment from the new display lens to the old display lens at the current color temperature value. This target conversion matrix reflects the difference in color sensitivity between the new and old display lenses. Then, a first color difference matrix is ​​determined between the first color matrix of the first image captured by the old display lens and the first AWB white point of the first image, taking into account the color difference between the first image captured by the old display lens and the first AWB white point. Based on the target conversion matrix, the first color difference matrix is ​​converted into a second color difference matrix corresponding to the new display lens. This ensures that the color difference between the second image captured by the new display lens and the second AWB white point of the new display lens remains consistent with the aforementioned color difference of the old display lens, effectively avoiding the color sensitivity difference between the new and old display lenses. This makes the calculated second AWB white point more accurate, and by using the second AWB white point to perform AWB processing on the image captured by the new display lens, the color difference between the images before and after lens switching can be reduced.

[0196] It should be noted that, in Figure 7In the process shown, taking the first screen and the second screen as examples of including a set of rectangular areas, in actual implementation, there are also cases where multiple sets of rectangular areas are included. That is, the first screen includes multiple first rectangular areas, and correspondingly, the second screen includes a second rectangular area corresponding to each first rectangular area.

[0197] Based on this, such as Figure 8 As shown, the method includes the following steps:

[0198] S801-S803 are the same as S701-S703 mentioned above.

[0199] S804. Calculate the first color matrix of each first rectangular region in the first frame captured by lens A.

[0200] S805, Calculate the second color matrix for each second rectangular region in the second frame captured by lens B.

[0201] S806, Obtain the first AWB white point of the first frame of lens A.

[0202] S807. Calculate the first color difference matrix between each first color matrix and the first AWB white point.

[0203] S808: Based on the target transformation matrix, each first color difference matrix is ​​converted into a second color difference matrix of the B lens.

[0204] In other words, each set of rectangular regions corresponds to a first color matrix, a second color matrix, a first color difference matrix, and a second color difference matrix.

[0205] S809. Based on the second color difference matrix and the second color matrix corresponding to each group of rectangular areas, obtain the first third AWB white point of the second image.

[0206] The method for calculating each third AWB white point in this step is the same as the method for calculating the second AWB white point in S709. Please refer to the relevant description in S709. It will not be repeated here.

[0207] Understandably, in this step, a third AWB white point can be calculated for each group of rectangular regions.

[0208] S810. Determine the distance between the position of the point represented by the first color matrix corresponding to each group of rectangular regions on the two-dimensional plane of R / B and B / G and the position of the first AWB white point.

[0209] S811. For each group of rectangular regions, determine the weight of the third AWB white point corresponding to that group of rectangular regions based on the distance to that group of rectangular regions.

[0210] Among them, there is a positive correlation between distance and the weight of the third WAB white point.

[0211] This application provides the following two methods for determining weights.

[0212] Method 1: For each group of rectangular regions, find the weight of the distance mapping corresponding to the group of rectangular regions from the preset mapping relationship between distance and weight, and use it as the weight of the third AWB white point corresponding to the group of rectangular regions.

[0213] As an example, the preset mapping relationship is shown in Table 1.

[0214] Table 1

[0215] distance Weight <0.1 1.0 0.1～0.2 0.5 0.2～0.4 0.3 0.4～0.8 0.2 >0.8 0.1

[0216] For example, if there are 3 sets of rectangular regions, and the distances between the R / B and B / G positions in the first color matrix of each set of rectangular regions and the first AWB white point position are 0.05, 0.25, and 0.7 respectively, then as shown in Table 1, the weights of the third AWB white points Prediction_WP_rgbg_ratio_B_1, Prediction_WP_rgbg_ratio_B_2, and Prediction_WP_rgbg_ratio_B_3 in each set of rectangular regions are 1.0, 0.3, and 0.2 respectively.

[0217] Method 2: Determine the sum of distances corresponding to each group of rectangular areas to obtain the total distance; for each group of rectangular areas, use the wallpaper with the first difference and the total distance as the weight of the third AWB white point corresponding to that group of rectangular areas.

[0218] The first difference is the sum of distances minus the product of the total number of rectangular regions and the distance corresponding to that rectangular region.

[0219] Suppose there are n sets of rectangular regions, and the distances corresponding to each set of rectangular regions are D1 to Dn, and the sum of D1 to Dn is D.

[0220] Then the weight W1 of the third AWB white point corresponding to the first rectangular region is (D – n * D1) / D;

[0221] The weight of the third AWB white point corresponding to the second rectangular region is W2 = (D – n * D2) / D;

[0222] The weight of the third AWB white point corresponding to the nth rectangular region is Wn = (D – n * Dn) / D.

[0223] It should be noted that if the weight of any set of rectangular regions is less than 0, the weight of that set of rectangular regions can be reset to 0.

[0224] S812. Based on the weight of each third AWB white point, perform a weighted summation of the third AWB white points to obtain the second AWB white point.

[0225] For example, suppose there are n groups of rectangular regions, and the third AWB white points corresponding to each group of rectangular regions are Prediction_WP_rgbg_ratio_B_1, Prediction_WP_rgbg_ratio_B_2, ..., Prediction_WP_rgbg_ratio_B_n, and the weights of the third AWB white points corresponding to each group of rectangular regions are W1, W2, ..., Wn, then the second AWB white point is (Prediction_WP_rgbg_ratio_B_1*W1+Prediction_WP_rgbg_ratio_B_2*W2+...+Prediction_WP_rgbg_ratio_B_n*Wn) / (W1+W2+...+Wn).

[0226] S813. Use the second AWB white point to perform AWB processing on the image captured by lens B.

[0227] Using this method, when calculating the second AWB white point of the new display lens, multiple third AWB white points of the new display lens can be calculated using multiple sets of rectangular regions in the first and second images. These third AWB white points are then weighted and summed to obtain the second AWB white point. Because the shorter the distance between the position of the point represented by R / B and B / G in the first rectangular region of the old display lens and the position of the first AWB white point when determining the weight of each third AWB white point, the smaller the error caused by the AWB white point conversion between the old and new display lenses through the target transformation matrix. Therefore, the weight of the third AWB white point corresponding to this first rectangular region is higher. Using this principle, the second AWB white point obtained by weighted summation has a smaller error, which can further improve the accuracy of the calculated second AWB white point, thereby reducing the color difference between the images before and after lens switching.

[0228] In a specific implementation, this application also provides a computer storage medium, including a computer program, wherein when the computer program runs on an electronic device, it causes the electronic device to execute some or all of the steps in the above embodiments. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0229] In a specific implementation, this application also provides a computer program product, which includes computer program code. When the computer program code is run on an electronic device, it causes the electronic device to perform some or all of the steps in the above method embodiments.

[0230] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0231] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0232] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0233] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image processing method based on white balance, characterized in that, include: When switching from an old display lens to a new display lens, the current color temperature value of the current scene captured by the old display lens is determined, the first color matrix of the first image of the current scene captured by the new display lens is determined, and the second color matrix of the second image of the current scene captured by the old display lens is determined. Obtain the preset color conversion matrix corresponding to the target standard light source that matches the current color temperature value. The preset color conversion matrix is ​​the conversion matrix for color alignment from the new display lens to the old display lens under the target standard light source. Based on the color temperature value corresponding to the target standard light source and the preset color conversion matrix, a target conversion matrix is ​​generated for color alignment from the new display lens to the old display lens under the current color temperature value. Determine the first color difference matrix between the first color matrix and the first AWB white point of the first image; Based on the target conversion matrix, the first color difference matrix is ​​converted into the second color difference matrix corresponding to the new display lens; Based on the second color difference matrix and the second color matrix, the second AWB white point of the second image is obtained; The image captured by the new display lens is processed using the second AWB white dot.

2. The method according to claim 1, characterized in that, Determining the first color matrix of the first image captured by the new display lens of the current scene, and the second color matrix of the second image captured by the old display lens of the current scene, includes: Obtain the first R / G value and the first B / G value of the first rectangular region in the first image. The first R / G value is the ratio between the average R value and the average G value of the first rectangular region in the first image. The first B / G value is the ratio between the average B value and the average G value of the first rectangular region in the first image. The first R / G value and the first B / G value are combined to form the first color matrix; The second R / G value and the second B / G value of the second rectangular region in the second image are obtained. The second rectangular region and the first rectangular region contain images of the same object. The second R / G value is the ratio between the average R value and the average G value of the second rectangular region in the second image. The second B / G value is the ratio between the average B value and the average G value of the second rectangular region in the second image. The second R / G value and the second B / G value are combined to form the second color matrix.

3. The method according to claim 1, characterized in that, The step of obtaining the preset color conversion matrix corresponding to the target standard light source matching the current color temperature value includes: Obtain the color temperature values ​​of each standard light source; If the current color temperature value is greater than the highest color temperature value among the color temperature values ​​of each standard light source, then the standard light source corresponding to the highest color temperature value is taken as the target standard light source; If the current color temperature value is less than the lowest color temperature value among the color temperature values ​​of each standard light source, then the standard light source corresponding to the lowest color temperature value shall be used as the target standard light source. If the current color temperature value is between the lowest color temperature value and the highest color temperature value, then select the first color temperature value and the second color temperature value that are adjacent to the current color temperature value from the color temperature values ​​of each standard light source, and use the first standard light source corresponding to the first color temperature value and the second standard light source corresponding to the second color temperature value as the target standard light source. Obtain the preset color conversion matrix corresponding to the target standard light source.

4. The method according to claim 3, characterized in that, The step of generating a target conversion matrix for color alignment from the new display lens to the old display lens under the current color temperature value, based on the color temperature value corresponding to the target standard light source and the preset color conversion matrix, includes: If the target standard light source includes one standard light source, then the preset color conversion matrix corresponding to the target standard light source is used as the target conversion matrix; If the target standard light source includes a first standard light source corresponding to the first color temperature value and a second standard light source corresponding to the second color temperature value, then the target conversion matrix is ​​obtained by interpolating the preset color conversion matrix corresponding to the first standard light source and the preset color transformation matrix corresponding to the second standard light source using the current color temperature value, the first color temperature value, and the second color temperature value.

5. The method according to claim 1, characterized in that, Determining the first color difference matrix between the first color matrix and the first AWB white point of the first image includes: The difference between the first color matrix and the first AWB white point is used as the first color difference matrix; The step of converting the first color difference matrix into the second color difference matrix corresponding to the new display lens based on the target conversion matrix includes: Multiply the first color difference matrix by the target transformation matrix to obtain the second color difference matrix; The step of obtaining the second AWB white point of the second image based on the second color difference matrix and the second color matrix includes: The difference between the second color matrix and the second color difference matrix is ​​used as the second AWB white point.

6. The method according to claim 2, characterized in that, The first screen and the second screen include multiple sets of rectangular areas. Each set of rectangular areas includes a first rectangular area in the first screen and a second rectangular area in the second screen. Each set of rectangular areas corresponds to a first color matrix, a second color matrix and a second color difference matrix. The step of obtaining the second AWB white point of the second image based on the second color difference matrix and the second color matrix includes: Based on the second color difference matrix and the second color matrix corresponding to each group of rectangular regions, a third AWB white point is obtained for the second image. Determine the distance between the position of the point represented by the first color matrix corresponding to each group of rectangular regions on the two-dimensional plane of R / B and B / G and the position of the first AWB white point; For each group of rectangular regions, the weight of the third AWB white point corresponding to the group of rectangular regions is determined based on the distance corresponding to the group of rectangular regions. There is a positive correlation between the distance and the weight of the third AWB white point. Based on the weight of each third AWB white point, the weighted sum of each third AWB white point is obtained to obtain the second AWB white point.

7. The method according to claim 6, characterized in that, The step of determining the weight of the third AWB white point corresponding to each group of rectangular regions based on the distance to that group of rectangular regions includes: For each group of rectangular regions, find the weight of the distance mapping corresponding to that group of rectangular regions from the preset mapping relationship between distance and weight, and use it as the weight of the third AWB white point corresponding to that group of rectangular regions; or, Determine the sum of the distances corresponding to each group of rectangular regions to obtain the total distance; For each group of rectangular regions, the ratio of the first difference to the sum of the distances is used as the weight of the third AWB white point corresponding to that group of rectangular regions. The first difference is the sum of the distances minus the product of the total number of groups of rectangular regions and the distance corresponding to that group of rectangular regions.

8. The method according to claim 1, characterized in that, The preset color conversion matrix corresponding to the target standard light source is obtained through the following steps: The first RAW image obtained by taking a picture of a preset standard color chart under the target standard light source using the old display lens; The second RAW image is obtained by taking a picture of the preset standard color chart under the target standard light source using the new display lens; Calculate the R / G value and B / G value of each color patch in the first RAW image to obtain the first matrix; Calculate the R / G value and B / G value for each color patch included in the second RAW image; The second matrix is ​​obtained; Subtract the R / G value of the reference color block in the first RAW image from each R / G value included in the first matrix, and subtract the B / G value of the reference color block in the first RAW image from each B / G value included in the first matrix to obtain the color characteristic matrix of the old display lens. The reference color block is a preset colorless color block in the preset standard color chart. Subtract the R / G value of the reference color block in the second RAW image from each R / G value included in the second matrix, and subtract the B / G value of the reference color block in the second RAW image from each B / G value included in the second matrix to obtain the color characteristic matrix of the new display lens; Based on the color characteristic matrix of the old display lens and the color characteristic matrix of the new display lens, a color conversion matrix for color alignment between the new display lens and the old display lens is calculated to obtain the preset color conversion matrix.

9. The method according to claim 8, characterized in that, The process of calculating a color conversion matrix based on the color characteristic matrix of the old display lens and the color characteristic matrix of the new display lens to align the colors from the new display lens to the old display lens, thereby obtaining the preset color conversion matrix, includes: The preset color conversion matrix is ​​calculated using the following formula: The preset color conversion matrix = inv(transpose of the color characteristic matrix of the new display lens * color characteristic matrix of the new display lens) * transpose of the color characteristic matrix of the new display lens * color characteristic matrix of the old display lens, where inv represents obtaining the inverse matrix.

10. An electronic device, characterized in that, include: One or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, Includes a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 9.

Images