Image processing method and electronic device

Abstract

The application provides an image processing method and an electronic device, and relates to the technical field of terminals. The application introduces semantic information and light source information to jointly perform partitioning, determines a more accurate partitioning result, adjusts an image by using multiple light sources, realizes local correction of a complex mixed light source scene, and improves image processing quality and an application range. The method comprises the following steps: detecting a user instruction shooting operation, collecting an initial image, and obtaining a first image and a second image from the initial image according to a reference image; the first image comprises a semantic partitioning result of the initial image, and the second image comprises a light source partitioning result of the initial image; obtaining a third image according to the first image and the second image; obtaining a light source estimation result according to the reference image and the first image; adjusting the color of the initial image according to the third image and the light source estimation result to obtain a target image; and displaying the target image.

Description

[0001] This application is a divisional application. The original application has the application number 202511336763.8 and the original application date is September 17, 2025. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to an image processing method and an electronic device. Background Technology

[0003] During the shooting process, the electronic device adjusts the color gain of the captured image, so that the image content can present more realistic colors under various lighting conditions, providing users with a better viewing experience.

[0004] For example, an electronic device is configured with a color correction model. This model segments the initial image into multiple smaller images of the same size, and uses these smaller images as auxiliary information to input into a convolutional neural network for feature extraction, thereby enhancing features in non-semantic regions. Subsequently, a global white balance algorithm is used to perform color correction on the enhanced image.

[0005] However, in real-world shooting environments, multiple light sources may exist, and global white balance algorithms cannot account for the differences in light sources in different areas, leading to significant color distortion in localized areas of the image. To address this, local white balance algorithms have been proposed; however, current local white balance algorithms suffer from inaccurate light source partitioning, affecting image processing quality. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides an image processing method and an electronic device. The technical solution provided by this application, by introducing semantic information and light source information to perform partitioning, can determine more accurate partitioning results and utilize multiple white points to precisely adjust the image locally, achieving local correction of complex mixed light source scenes, thereby improving image processing quality and expanding its applicability.

[0007] To achieve the above-mentioned technical objectives, this application provides the following technical solution: In a first aspect, an image processing method is provided, applied to an electronic device. The method includes: detecting a user-instructed shooting operation; acquiring an initial image and a reference image; acquiring a first image and a second image based on the initial image; wherein the first image includes a semantic partitioning result of the initial image, and the second image includes a light source partitioning result of the initial image; acquiring a third image based on the first image and the second image; acquiring a light source estimation result based on the reference image and the first image; adjusting the color of the initial image based on the third image and the light source estimation result to acquire a target image; and displaying the target image.

[0008] In some examples, the third image is a grayscale image obtained by fusing the first and second images. The initial and reference images are both images in raw image file format, which are raw data sets obtained without any processing.

[0009] Thus, the light source information in the initial image can be accurately identified based on the third image. Furthermore, by introducing semantic information for partitioning, subsequent image processing operations (such as white balance adjustment) are no longer global adjustments but can be precisely and locally performed, thereby achieving local correction for complex mixed light source scenes. Even when the semantic information in the initial image is uncertain, it can be protected by separate partitioning, avoiding incorrect color processing of the semantic information and improving generalization. Since the output first image is relatively stable and has high accuracy and robustness, the accuracy of the third image obtained by fusing the first and second images is improved, resolving the problem of light source ambiguity.

[0010] According to the first aspect, obtaining a light source estimation result based on a reference image and a first image includes: in the case where there is a relative angle deviation between the initial image and the reference image, obtaining the light source estimation result after aligning the reference image and the first image according to the relative angle deviation.

[0011] According to the first aspect, or any implementation of the first aspect above, after aligning the reference image and the first image according to the relative angle deviation, obtaining the light source estimation result includes: aligning the first image with the reference image according to the relative angle deviation; and obtaining the light source estimation result based on the aligned first image and the reference image.

[0012] For example, using a reference image as a standard, the angular deviation between the reference image and the first image is determined. The angle of the first image is then adjusted to match the angle of the reference image based on this angular deviation, resulting in an aligned first image. For instance, the relative angular deviation between the two images can be determined based on the relative displacement deviation of corresponding pixels between the reference image and the first image.

[0013] Thus, because the distinction between the person and the background is relatively clear in the third image, the spatial features of the foreground person and the background are processed separately based on the third image during light source estimation. This eliminates the ambiguity of light sources caused by different light sources in space, making the light source estimation results more accurate and stable. This solves the problems of color cast and flickering in the person and background areas in real-world scenes. It also achieves more accurate color reproduction in photography scenarios, improving the precision of the algorithm results.

[0014] According to the first aspect, or any implementation of the first aspect above, after aligning the reference image and the first image according to the relative angle deviation, the light source estimation result is obtained, including: aligning the reference image with the initial image according to the relative angle deviation; and obtaining the light source estimation result based on the aligned reference image and the first image.

[0015] For example, the initial image is used as a standard. Since the first image is generated based on the initial image, the first image can also be used as a standard. The angular deviation between the initial image and the reference image is determined, and the angle of the reference image is adjusted to the angle of the initial image according to the angular deviation to obtain the aligned reference image.

[0016] In this way, by aligning the reference image with the initial image as the standard, the spatial features of the foreground portrait and the background are processed separately based on the third image and the aligned reference image when estimating the light source. This eliminates the ambiguity of the light source caused by different light sources in space, making the light source estimation results more accurate and stable.

[0017] According to the first aspect, or any implementation of the first aspect above, the first image is obtained based on the initial image, including: based on the offset angle information of the reference image, the first image aligned with the reference image is obtained based on the initial image.

[0018] In this way, the first image aligned with the reference image is obtained directly from the reference image and the initial image, which improves the generation efficiency of the first image.

[0019] According to the first aspect, or any implementation of the first aspect above, the light source estimation result is obtained based on the reference image and the first image, including: obtaining light source information corresponding to the reference image; and obtaining the light source estimation result based on the first image and the light source information.

[0020] In this way, a reference image with more channels can be used as auxiliary information in the light source determination process. Even if the specific information of the first image cannot be determined, the light source information can be determined more accurately based on the reference image, and then more accurate dual-light source information can be determined. This is not limited by the accuracy of semantic recognition and can be applied on a large scale.

[0021] According to the first aspect, or any implementation of the first aspect above, the light source estimation result is obtained based on the first image and the light source information, including: obtaining the light source estimation result of the target number based on the first image and the light source information.

[0022] In this way, the estimation results of multiple light sources can be used to make precise and local adjustments, thereby achieving local correction of complex mixed light source scenes.

[0023] According to the first aspect, or any implementation of the first aspect above, the light source estimation result indicates information about at least two light sources included in the reference image.

[0024] According to the first aspect, or any implementation of the first aspect above, the electronic device includes at least one camera, wherein the initial image and the reference image are images captured by the same camera, or images captured by different cameras.

[0025] According to the first aspect, or any implementation of the first aspect above, the initial image and the reference image are the same image, or the initial image and the reference image are different images.

[0026] According to the first aspect, or any implementation of the first aspect above, when the initial image and the reference image are different images, the number of channels of the initial image is less than the number of channels of the reference image.

[0027] According to the first aspect, or any implementation of the first aspect above, when the initial image and the reference image are different images, the resolution of the initial image is greater than the resolution of the reference image.

[0028] Thus, because the reference image has a large number of channels, the electronic device can determine the light source more accurately based on the number of channels in the reference image, and obtain a more accurate dual-light source estimation result.

[0029] In a second aspect, an electronic device is provided. The electronic device includes: a processor and a memory, the memory being coupled to the processor. The memory stores computer program code, which includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device performs the following actions: detecting a user-instructed action to capture an image; acquiring an initial image and a reference image; acquiring a first image and a second image based on the initial image; wherein the first image includes a semantic partitioning result of the initial image, and the second image includes a light source partitioning result of the initial image; acquiring a third image based on the first image and the second image; acquiring a light source estimation result based on the reference image and the first image; adjusting the color of the initial image based on the third image and the light source estimation result to acquire a target image; and displaying the target image.

[0030] According to the second aspect, when the processor reads computer instructions from memory, it also causes the electronic device to perform: in the case of a relative angular deviation between the initial image and the reference image, after aligning the reference image and the first image according to the relative angular deviation, to obtain the light source estimation result.

[0031] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to perform: aligning the first image with the reference image based on the relative angle deviation; and obtaining a light source estimation result based on the aligned first image and the reference image.

[0032] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to perform: aligning the reference image with the initial image based on the relative angle deviation; and obtaining the light source estimation result based on the aligned reference image and the first image.

[0033] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to perform: based on the offset angle information of the reference image, acquire a first image aligned with the reference image according to the initial image.

[0034] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to perform: acquiring light source information corresponding to the reference image; and acquiring a light source estimation result based on the first image and the light source information.

[0035] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to perform: obtaining a light source estimation result for the number of targets based on the first image and light source information.

[0036] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to execute: the light source estimation result indicates information about at least two light sources included in the reference image.

[0037] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to execute: the electronic device includes at least one camera, and the initial image and the reference image are images captured by the same camera, or images captured by different cameras.

[0038] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to execute: the initial image is the same as the reference image, or the initial image is different from the reference image.

[0039] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to execute: in the case that the initial image and the reference image are different images, the number of channels of the initial image is less than the number of channels of the reference image.

[0040] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to execute: in the case that the initial image and the reference image are different images, the resolution of the initial image is greater than the resolution of the reference image.

[0041] Thirdly, an electronic device is provided, which has the function of implementing the image processing method as described in the first aspect and any of its possible implementations. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described function.

[0042] Fourthly, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program (also referred to as instructions or code) that, when executed by an electronic device, causes the electronic device to perform the method of the first aspect or any embodiment of the first aspect.

[0043] Fifthly, a computer program product is provided that, when run on an electronic device, causes the electronic device to perform the method of the first aspect or any one of the embodiments of the first aspect.

[0044] In a sixth aspect, a circuit system is provided, the circuit system including processing circuitry configured to perform the method of the first aspect or any embodiment of the first aspect.

[0045] In a seventh aspect, a chip system is provided, including at least one processor and at least one interface circuit, wherein the at least one interface circuit is used to perform transceiver functions and send instructions to the at least one processor, and when the at least one processor executes the instructions, the at least one processor performs the method of the first aspect or any embodiment of the first aspect.

[0046] The technical effects of the aforementioned aspects can be referenced from each other, and will not be elaborated further here. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of an image processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of image color block extraction provided in an embodiment of this application; Figure 3 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application; Figure 4 A schematic flowchart of an image processing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of an image partitioning process provided in an embodiment of this application; Figure 6 A flowchart illustrating image light source estimation provided in this application embodiment. Figure 1 ; Figure 7 A flowchart illustrating image light source estimation provided in this application embodiment. Figure 2 ; Figure 8 A schematic diagram illustrating an image alignment method provided in an embodiment of this application; Figure 9 A flowchart illustrating an image processing method provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one or more (including two).

[0049] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0050] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0051] In some embodiments, image acquisition devices such as cameras and mobile phones aim to obtain image information that is color-accurate and can reproduce the real scene during the image acquisition process. However, since the image signal processor (ISP) set in the acquisition device does not have the intelligent color adjustment ability of the human brain, it is easily affected by the lighting conditions in the shooting environment during the image acquisition process, resulting in color deviation in the final image, that is, color cast or white balance imbalance.

[0052] Optionally, white balance (WB) is a key functional module of the image signal processor, designed to ensure that white objects appear true white under various lighting conditions. White balance enables the image acquisition device to identify white in the current scene and, using this identified white as a standard, corrects the colors of all other objects in the scene by correcting color deviations caused by the color temperature of the light source, thereby presenting an image with true colors as perceived by the human eye under different light sources. Optionally, white balance processing typically involves two steps: light source information estimation and color adaptation compensation. Light source information estimation determines the actual ambient light source at the time of shooting based on data captured by the color image sensor set in the image signal processor. Color adaptation compensation predicts and simulates the human eye and brain's ability to correct color, ensuring that images of an object taken under different lighting conditions display the colors perceived by the human eye on different image acquisition devices.

[0053] In some embodiments, traditional white balance algorithms rely on prior assumptions about the physical world. For example, before processing an image, the algorithm of the built-in image processing model does not correct colors by analyzing and recognizing scene content, but rather forcibly assumes that the scene conforms to certain statistical or physical laws, and performs mathematical calculations based on this assumption and a pre-set color correction model. For example, it assumes the light source information of the scene, and then performs color adaptation compensation on the image based on this assumed light source information.

[0054] In some embodiments, traditional white balance algorithms typically require a large scene coverage area for effective global light source prediction. This method can effectively correct overall color deviation under conditions of a single light source and rich scene content. However, actual shooting scenes are usually multi-light source scenes. In such cases, if color adaptation compensation is performed according to a single light source environment, it will cause color deviation in areas affected by the second light source. Therefore, traditional global white balance algorithms cannot take into account the differences in light sources in different areas, resulting in color distortion in local areas of the image. For example, if an image is taken indoors near a window, the image contains both indoor and outdoor light sources. Since traditional white balance algorithms determine a single light source based on the scene coverage area, if the indoor environment of the image is large, the indoor light source will be determined as the only light source in the image. During color correction, because the outdoor light source is different from the indoor light source, color adaptation compensation based on the indoor light source will cause color deviation in the area covered by the outdoor light source.

[0055] Optionally, existing image color correction models typically perform color correction based on traditional global white balance algorithms, which are implemented using white points in the image. When using global white balance algorithms, a white point is usually set for the acquired image based on color constancy, and then color correction is performed based on this white point, which can adjust the entire image area. Color constancy means that even if the color of the light illuminating an object changes (e.g., color temperature), the object's color is still perceived as its true color. Color is an inherent attribute of the object, not a temporary effect under lighting. However, if the acquired image is from a multi-light source scene with multiple areas of local color cast, setting a single white point only allows for global white balance adjustment and cannot address the issue of semantic information color changes under different light source conditions, thus failing to meet current color correction requirements.

[0056] Optionally, the semantic information of an image refers to the meaning, content, story, emotion, and intention it conveys. For example, an image contains directly identifiable features such as pixel color (e.g., color standard (red, green, blue, RGB) values), edges, texture, and shape. Semantic information is the object information, scene information, and conceptual information of the image obtained after analyzing these features.

[0057] In some embodiments, to address the issue of color distortion in local image regions, a local white balance algorithm has been proposed for image color correction in multi-light source scenes. The local white balance algorithm first distinguishes the areas affected by different light sources in the scene, and then determines different white points in these different light source-affected areas.

[0058] Optionally, the white point refers to the color value of an object that is perceived as truly white by human vision under the current lighting conditions. For example, in an indoor environment illuminated by warm yellow tungsten lamps, a white object may appear yellowish and have a lower color temperature when displayed in an image. In this case, the yellowish color value of the white object displayed in the image is defined as the white point, also called the estimated white point. The yellowish color value then needs to be corrected to a color value without color cast. First, the color value of a white object under a color cast condition is obtained, and this color value is used as the reference point for color correction, i.e., the target white point. Then, color correction is performed on the area corresponding to the estimated white point based on the value of the target white point.

[0059] Optionally, errors in determining the white point can cause an overall color cast of a particular light source. Incorrectly defining the area affected by the light source can lead to incorrect affected areas, resulting in various problems such as localized color casts, unnatural transitions, and color jumps between video frames. For example, targets like portraits might be incorrectly classified as areas affected by the light source. Therefore, the accuracy of defining the area affected by the light source and determining the white point directly impacts the image display.

[0060] For example, such as Figure 1 As shown, the color correction model includes a semantic constraint module and an auxiliary calibration module. The semantic constraint module inputs the acquired initial image into the first convolutional neural network for semantic feature extraction, thereby obtaining the features of the initial image. Figure 1 Among them, features Figure 1 This is a semantic feature map containing high-level semantic information. The initial image is then input into a second convolutional neural network for feature extraction, yielding the features of the initial image. Figure 2 The information output from the second convolutional neural network is then input into the convolutional layer to obtain the output of the convolutional layer. Features are then calculated using a similarity loss function. Figure 1 With features Figure 2 The similarity loss value expands the feature representation of the feature map from simple color features to a hybrid feature combining color and semantics. The auxiliary calibration module segments the acquired initial image into multiple smaller images of the same size. These smaller images are used as auxiliary information and input into the third convolutional neural network for feature extraction. The feature information output by the third convolutional neural network is then input into the convolutional layer to obtain the prediction result, thereby enhancing the features of non-semantic regions. However, although this method can achieve the effect of image feature enhancement, the image color correction still relies on global white balance and color constancy after feature enhancement. It cannot accurately identify changes in semantic information color under different lighting conditions. Therefore, the adjusted image cannot achieve more accurate local white balance correction.

[0061] In some embodiments, the color correction model utilizes a multi-source color constancy network capable of color decoupling to analyze and identify the acquired image, determining the light source information and semantic content within the image and partitioning it. Then, the color correction model uses a contrast coordination module to decouple the light source colors in the image, identifying the light source information for each region of the image and performing color correction based on the different light source information. However, this approach requires accurate light source partitioning for each acquired image, which is difficult to acquire and prone to semantic information misidentification, making large-scale execution impossible.

[0062] For example, such as Figure 2 As shown, after acquiring the initial image, the color correction model extracts its semantic information and collects different pixel colors from the initial image. It then compares the collected pixel colors with colors in a color wheel containing multiple colors to determine their similarity. Based on the semantic information and similarity scores, it determines whether the pixel color corresponds to a material color or a light source color. The model then divides the light source-affected region based on the semantic information and pixel color. However, this approach requires understanding the color constancy of objects in the initial image. If the objects in the image are unfamiliar figures, objects, or scenes, it is difficult to determine whether the object's pixel color is a material color or a light source color, making accurate light source-affected region division difficult and resulting in poor adaptability.

[0063] For example, the color correction model obtains an initial image that is a multi-light source image under complex conditions, such as... Figure 2 As shown in (a), when an image contains objects of different colors such as blue and yellow, the color correction model tends to identify yellow objects as objects illuminated by warm light sources and blue objects as objects illuminated by cool light sources. In scenarios where the material color of an object is identified as the color of the light source, misclassification can occur, for example... Figure 2 As shown in (b), the blue object is under warm light, but because its material color is blue, it is mistakenly classified as being in the influence area of ​​a cool light source. Similarly, the yellow object is under cool light, but because its material color is yellow, it is mistakenly classified as being in the influence area of ​​a warm light source. After the influence area of ​​an object is correctly defined, color correction is performed based on the light source. If the influence area of ​​an object is incorrectly defined, it can easily lead to incorrect color adjustments.

[0064] Optionally, the warm and cool color classification of a light source is determined based on its color temperature. Color temperature is a physical quantity that describes the color characteristics of a light source, measured in Kelvin (K). It refers to the color of light emitted by a black body when heated to a specific temperature. A black body is an ideal object that does not reflect any light at all. When it is continuously heated, it emits different colors of light as its temperature increases.

[0065] Optionally, the lower the K value of the color temperature, the warmer the light (leaning towards reddish-yellow); the higher the K value, the cooler the light (leaning towards bluish). For example, a color temperature of 2000K is considered warm light, and a color temperature of 8000K is considered cool light. The color temperature range generally considered to be neutral white is approximately 4500K-5500K. For example, a color temperature below 4500K is considered warm light, and a color temperature above 5500K is considered cool light.

[0066] For example, such as Figure 2 As shown in (a), the initial image acquired by the color correction model is an indoor scene. In such scenarios, accurate light source information can be obtained by simulating the light source in the image using various techniques, such as simulating real-world lighting or using image processing software for virtual lighting simulation. However, in practical applications, if the initial image received by the color correction model is an outdoor scene or another scene where lighting simulation is difficult, and the model has never been trained on such a scene, accurate light source information cannot be obtained. Furthermore, during the training process, it is difficult to use images with all lighting conditions as a training set for the color correction model. Therefore, this method has limited versatility and cannot adapt to complex environmental scenarios.

[0067] Based on this, this application provides an image processing method that combines semantic information with the division of light source influence areas to achieve accurate identification of semantic information in the image and obtain accurate light source partitioning results. Then, corresponding white points are set for color adjustment in different light source partitioning results, improving the accuracy and applicability of image color adjustment in mixed light source scenes.

[0068] In some embodiments, the image processing method provided in this application can be applied to electronic device 100. In some examples, electronic device 100 includes, but is not limited to, mobile phones, cameras, tablets, wearable devices (such as smartwatches, smart bracelets, etc.), laptops, smart cars, robot vacuum cleaners, displays, ultra-mobile personal computers (UMPCs), smart screens, netbooks, personal digital assistants (PDAs), artificial intelligence (AI) devices, augmented reality (AR) / virtual reality (VR) glasses, and other terminal devices with shooting and recording functions. The operating system installed on electronic device 100 includes, but is not limited to, iOS®, Android®, HarmonyOS®, Windows®, Linux®, or other operating systems. This application does not limit the specific type of electronic device 100 or the operating system installed on it.

[0069] Optionally, the electronic device 100 in this application embodiment can be implemented by different devices. Different devices can have the same, similar, or somewhat different hardware structures, such as... Figure 3 The diagram shows the hardware structure of the electronic device.

[0070] For example, electronic device 100 has such Figure 3 Taking the hardware structure shown as an example, for Figure 3 The hardware structure shown will be explained.

[0071] like Figure 3 As shown, the electronic device 100 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a sensor module 380, a button 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a subscriber identification module (SIM) card interface 395, etc.

[0072] Optionally, the number of cameras 393 is not limited. If there are multiple cameras, for example, camera 393 includes a main camera and a secondary camera. The main camera is used for imaging, and the image captured by the main camera has three channels in a color space (red, green, blue, RGB), similar to human vision. The secondary camera is a reference camera that is always on; it only provides data as an auxiliary camera and does not participate in imaging. Optionally, there can be at least one main camera and at least one secondary camera. Optionally, the electronic device 100 uses the initial image captured by the main camera and the reference image captured by the secondary camera to output a target image with accurate color. The image data processing process is detailed below and will not be repeated here. Optionally, camera 393 may also include cameras other than the main and secondary cameras for implementing other functions. If there is only one camera, this camera can capture both the initial image and the reference image. Regardless of the number of cameras the electronic device has, the initial image and the reference image captured by the electronic device can be the same image, or they can be different images. Optionally, this solution aims to adjust the image color based on the images captured by the cameras. An electronic device includes at least one camera, which can capture the same or different initial images and reference images. If the electronic device includes multiple cameras, it includes at least one main camera and one secondary camera, and may include multiple secondary cameras in different situations. Typically, the image captured by the main camera has a higher resolution than the image captured by the secondary cameras. It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0073] Processor 310 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0074] In some embodiments, the processor 310 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0075] The MIPI interface can be used to connect the processor 310 to peripheral devices such as the display screen 394 and the camera 393. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 310 and the camera 393 communicate via the CSI interface. The processor 310 and the display screen 394 communicate via the DSI interface to realize the display function of the electronic device 100.

[0076] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0077] The mobile communication module 350 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 350 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 350 may be housed in the processor 310. In some embodiments, at least some functional modules of the mobile communication module 350 and at least some modules of the processor 310 may be housed in the same device.

[0078] The wireless communication module 360 ​​can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 310. The wireless communication module 360 ​​can also receive signals to be transmitted from processor 310, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0079] Electronic device 100 implements display functions through a GPU, display screen 394, and application processor. The GPU is a microprocessor for image processing, connecting the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0080] Display screen 394 is used to display images, videos, etc. Display screen 394 includes a display panel. The display panel can be manufactured using a liquid crystal display (LCD), such as an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 394, where N is a positive integer greater than 1.

[0081] The following section uses an electronic device that includes a main camera and a secondary camera, with the main camera capturing an initial image and the secondary camera capturing a reference image as an example, to provide a detailed description of the image processing method provided in the embodiments of this application.

[0082] In some embodiments, as described above, white balance processing typically includes two steps: light source information estimation and color adaptation compensation. Light source information estimation is used to determine the light source information present in the current shooting environment, such as the number of light sources. The electronic device can then perform light source partitioning and semantic partitioning on the initial image captured by the main camera, and based on the results of these two partitionings, achieve a more accurate determination of the light source information. Color adaptation compensation is used to determine white point information in the image, thereby compensating for image color based on the white point information. The electronic device can then determine the white point information using a reference image captured by the secondary camera. Subsequently, the electronic device can perform color compensation of the image based on the partitioning results and the white point information to output a color-accurate target image.

[0083] Optionally, during the process of obtaining partitioning results based on the initial image, the electronic device may also refer to the reference image to achieve more accurate partitioning results. Optionally, during the process of obtaining white point information based on the reference image, the electronic device may also refer to both the initial image and the partitioning results to achieve more accurate acquisition of white point information.

[0084] It should be understood that the above process of obtaining partitioning results and obtaining white point information can be executed in parallel.

[0085] Optionally, the electronic device uses a partitioning module to estimate the light source information of the initial image and a dual white point estimation model to achieve color adaptation compensation of the initial image.

[0086] For example, such as Figure 4 As shown, the electronic device inputs the initial image captured by the main camera into the partitioning module to obtain the partitioning result output by the partitioning module, such as a partitioned grayscale image. Optionally, the electronic device inputs the initial image captured by the main camera and the reference image captured by the secondary camera into the dual white point estimation model (i.e., the dual light source estimation model) to obtain white point information, such as the dual white point estimation result (i.e., the dual light source estimation result). Then, the electronic device inputs the initial image, the dual white point estimation result, and the partitioned grayscale image into the image adjustment module, which performs corresponding color adjustments to obtain a color-accurate target image.

[0087] The following section provides a detailed explanation of the process for acquiring the target image.

[0088] In some embodiments, the electronic device captures an initial image in raw image file format via a main camera, and a reference image corresponding to the initial image via a secondary camera.

[0089] Optionally, RAW format is the raw data set obtained after the image sensor converts the captured light signals into digital signals, without any processing. It is a data packet that records the most original and complete light information of the shooting scene. RAW format files record the intensity (i.e., brightness) of light perceived by each pixel in the initial image, rather than direct color information.

[0090] Optionally, the initial image has spatial resolution, which refers to the level of detail that can be distinguished in the image.

[0091] Optionally, the reference dimensions for spatial resolution are (H, W, C), where H > 1 and W > 1. Here, H is Height, representing the number of pixels in the vertical direction of the initial image; W is Width, representing the number of pixels in the horizontal direction of the initial image; and C is Channels, describing the number of information components required for each pixel. For example, common color images typically have three channels: red, green, and blue. The colors from these three channels are mixed together to determine the final color of the pixel. Grayscale images have only one channel, storing only brightness information. Portable network graphic format (PNG) images with transparency have four channels: red, green, blue, and alpha.

[0092] In some embodiments, the secondary camera is a camera that acquires multi-channel information, and the reference image obtained based on the multi-channel information can include more accurate spectral information. Therefore, the electronic device can more accurately determine the number of light sources corresponding to the reference image based on the spectral information. Optionally, the initial image acquired by the main camera and the reference image acquired by the secondary camera are images with different numbers of channels but the same or similar content. Therefore, the number of light sources corresponding to the reference image is also the number of light sources corresponding to the initial image. Thus, the electronic device can determine the number of light sources corresponding to the reference image through a preset method, and then determine the number of light sources corresponding to the initial image.

[0093] Optionally, preset methods may include determining the number of light sources based on the number of cluster centers, or determining the number of light sources based on color shift.

[0094] In some examples, the electronic device projects the color values ​​(i.e., spectral information) of all pixels in a reference image into a high-dimensional color space, and then uses a clustering algorithm to cluster the pixel color values, with the cluster centers considered as potential light sources. Therefore, the number of cluster centers is determined as the number of light sources. Each cluster center represents the color distribution under a dominant light source.

[0095] Optionally, the electronic device acquires more new feature information corresponding to the color value, and determines the information composed of the color value and the new feature information as the information corresponding to the high-bit color space. The new feature information is discriminative, providing additional information that makes the originally mixed data separable, such as transparency, chroma, brightness, and saturation. Chroma includes, for example, the red-to-green ratio (R / G) and the red-to-blue ratio (R / B). If some pixels exhibit similar color values ​​in the spectral information, they are distinguished using the information corresponding to the high-bit color space. If these pixels with similar color values ​​show significant differences in the new feature information, then the light source information of these pixels can be more accurately distinguished based on the new feature information.

[0096] In other examples, the electronic device directly determines the reference image captured by the secondary camera as an image with a light source count of 1, and then performs global correction on the reference image to obtain a corrected reference image. The electronic device then determines whether color cast still exists in the corrected reference image. If color cast still exists in the corrected reference image, it determines that the number of light sources in the reference image captured by the secondary camera is greater than 1.

[0097] Optionally, color cast refers to an image's colors deviating from their proper, true, or expected colors, presenting an unnatural color bias.

[0098] In some embodiments, if the number of light sources in the initial image is 1, the electronic device determines that the initial image is a single-light source image. When performing color correction on the initial image, there is no need to partition the light sources; a global adjustment can be performed directly. If the number of light sources in the initial image is greater than 1, the electronic device determines that the initial image is a multi-light source image. When color correction is required on the initial image, the light source information in the initial image must first be determined, and the initial image must be partitioned by light sources.

[0099] Optionally, if the number of light sources in the initial image is greater than 1, the electronic device typically assumes that the number of light sources in the initial image is 2. Alternatively, in some complex scenarios, the number of light sources can be determined to be 3 or more. This application embodiment uses a number of light sources of 2 as an example to describe the image processing method provided by this application embodiment.

[0100] In some embodiments, after determining the number of light sources, if the number of light sources is greater than 1, the initial image is determined to be a multi-light source scene, which can easily lead to partitioning errors during the light source partitioning process. Therefore, a semantic partitioning module is added to the partitioning module of the electronic device to identify semantic information in the initial image for assistance, so as to accurately determine the light source information corresponding to the semantic information. Then, the partitioning module merges the output of the semantic partitioning with the output of the light source partitioning module set in the partitioning module to obtain the final accurate partitioning result of the initial image, that is, the partitioned grayscale image.

[0101] In some embodiments, the partitioning module in the electronic device inputs the initial image to the semantic partitioning module of the color correction model for semantic feature extraction, resulting in a semantic grayscale image (mask) containing semantic information from the initial image. For example, when the semantic partitioning model extracts portrait features, it obtains a portrait grayscale image (mask) containing portrait information from the initial image. The semantic partitioning module can extract arbitrary semantic features during feature extraction, such as portrait features, object features, scene features, etc.

[0102] Optionally, the electronic device includes a color correction model for modifying the acquired images. This model is deployed on the device's computing nodes and adapted to the hardware (such as an image sensor) through software modifications to improve image color processing performance. The computing nodes can be central processing units (CPUs), graphics processing units (GPUs), neural network processors, tensor processing units (TPUs), or application-specific integrated circuits (ASICs), etc.

[0103] Optionally, the semantic partitioning module identifies the portrait semantic information in the initial image, extracts the pixel information identified as portrait semantic information, and obtains the corresponding portrait grayscale image.

[0104] Optionally, after the initial image is input into the semantic partitioning module in the electronic device, the partitioning module determines the semantic information to be identified according to different recognition requirements, such as the semantic information being a human figure or an object.

[0105] Optionally, the feature extraction methods set in the semantic partitioning module and the light source partitioning module of the electronic device include, but are not limited to, methods based on deep learning model architectures. These deep learning model architecture-based methods include, for example, convolutional neural networks (CNNs) and self-attention mechanisms (transformers).

[0106] Thus, the grayscale image of the human figure originates from the output of the semantic partitioning module. Since the semantic partitioning module can acquire a large amount of training data through various means such as open-source data and ordinary cameras, the confidence of its output is improved after training with this acquired data. Therefore, the semantic partitioning module can achieve accurate and stable output results, resolving issues such as ambiguity in the light source of the human figure and flickering between video frames.

[0107] In some embodiments, the partitioning module in the electronic device inputs the initial image to the light source partitioning module of the color correction model. The light source partitioning module identifies the light source feature information in the initial image, outputs the gray value of the light source feature information corresponding to each pixel in the initial image, and then partitions the light source according to the output gray value to obtain a light source grayscale image (mask) containing the light source information in the initial image.

[0108] Optionally, the light source feature information in the initial image can be identified. For example, pixels with similar color temperatures in the initial image can be divided into the same region, light sources can be partitioned according to different objects or regions (such as sky, ground, etc.) in the initial image identified semantically, or light sources can be partitioned according to regions manually selected by the user.

[0109] For example, such as Figure 5As shown, the electronic device acquires an initial image (a) in its original image file format by capturing images through the main camera. Then, the electronic device can input the initial image (a) into the partitioning module. The semantic partitioning module within the partitioning module extracts features from the initial image (a), identifying region 01 as human portrait information. Therefore, region 01 is extracted from the initial image (a) to obtain the corresponding grayscale image of the human portrait (b). Optionally, the electronic device can also extract features from the initial image (a) through the light source partitioning module within the partitioning module, identifying region 02 as the ground, regions 03 and 04 as walls, region 05 as the indoor ceiling wall, and region 06 as a door. The light intensity of regions 02 to 06 is extracted, and corresponding light source grayscale images (c) are generated according to the different light intensities of each region.

[0110] Optionally, during the acquisition of the grayscale image of the portrait (b), the portrait is usually represented by a uniform color, such as black. In some cases, a grayscale image of the portrait with color gradients can also be generated based on the lighting conditions of the portrait portion of the actual original image.

[0111] Optionally, the light source grayscale image (c) output by the light source partitioning module is an image carrying pixel information of different brightness, that is, it can represent the transitional changes in the brightness of light in the initial image.

[0112] Optionally, a grayscale image (mask) is a grayscale image of the same size as the original image, used to selectively cover or display certain areas of the original image, thereby enabling precise manipulation of specific areas.

[0113] In this way, the light source information in the initial image can be accurately identified based on the grayscale image. Furthermore, by introducing semantic information for partitioning, subsequent image processing operations (such as white balance adjustment) are no longer global adjustments, but can be precisely and locally adjusted, thereby achieving local correction for complex mixed light source scenes. Even when the semantic information in the initial image is uncertain, the semantic information can be protected by separate partitioning, preventing global white balance from incorrectly processing the color of the semantic information and improving generalization.

[0114] In some embodiments, a light source grayscale image and a portrait grayscale image are input into the fusion module of the color correction model of an electronic device. The light source grayscale image and the portrait grayscale image are fused using a preset fusion method in the fusion module to obtain a partitioned grayscale image.

[0115] For example, such as Figure 5As shown, the partitioning module of the electronic device finally outputs a partitioned grayscale image (d). The partitioned grayscale image (d) contains a portrait grayscale image (b) and a light source grayscale image (c). It can accurately display the area of ​​light source information in the initial image and completely and accurately divide the area corresponding to the semantic information, which is beneficial for subsequent local white balance processing.

[0116] Optionally, the preset fusion methods include, but are not limited to, image domain filtering algorithms and methods based on deep learning model architectures. Image domain filtering algorithms include bilateral filtering algorithms, guided filtering algorithms, bilateral optimizer algorithms, etc. Methods based on deep learning model architectures include convolutional neural networks, self-attention mechanisms, etc.

[0117] Thus, since the output of the semantic partitioning module is relatively stable and has high accuracy and robustness, the accuracy of fusing the grayscale images output by the semantic partitioning module and the light source partitioning module to obtain the partitioned grayscale image is improved, and the problem of light source ambiguity is solved.

[0118] In some embodiments, since the confidence level of the semantic grayscale image is higher than that of the light source grayscale image, the resulting partitioned grayscale image after fusing the semantic grayscale image and the light source grayscale image achieves accurate and stable segmentation of semantic information, solving problems such as semantic information ambiguity and inter-frame flickering. Because a more accurately segmented partitioned grayscale image is obtained, different partitions can be selected for adjustment more accurately when performing image color adjustment based on the partitioned grayscale image.

[0119] The above section introduced the process of estimating light source information. The following section details the process of color adaptation compensation for the partitioned grayscale image determined based on the light source information estimation.

[0120] In some embodiments, during the dual white point estimation process, because the reference image has a larger number of channels, the electronic device can more accurately determine the light source based on the number of channels in the reference image, resulting in a more accurate dual white point estimation result. The dual white point estimation module maps the light source information of the reference image to the semantic grayscale image to determine the dual white point estimation result. At this time, if there is a viewing angle deviation between the reference image and the initial image, it indicates that there is also a viewing angle deviation between the semantic grayscale image and the reference image, and the light source information of the reference image cannot be accurately mapped. Therefore, before performing dual white point estimation, it is necessary to adjust the viewing angle between the semantic grayscale image and the reference image to be consistent. The semantic grayscale image, the reference image captured by the secondary camera, and the initial image captured by the main camera are input into the image alignment module in the color correction model of the electronic device to adjust the semantic grayscale image. The reference image is used as a standard, and then the angle of the semantic grayscale image is adjusted according to the reference image until the semantic grayscale image is aligned with the reference image, resulting in an aligned semantic grayscale image, which is then output.

[0121] Optionally, there may be differences in resolution and number of channels between the reference image and the initial image. The initial image acquired by the main camera is typically a three-channel image, but in some special cases it may be an image with a different number of channels. Generally, the reference image acquired by the secondary camera has a greater number of channels than the initial image acquired by the main camera, and the resolution of the reference image acquired by the secondary camera is lower than the resolution of the initial image acquired by the main camera.

[0122] For example, such as Figure 6 As shown, Figure 6 (a) in the image is the initial image captured by the main camera. After inputting the initial image captured by the main camera into the semantic partitioning module in the partitioning module, the result is as follows: Figure 6 The semantic grayscale image shown in (b) above. For example, refer to the above. Figure 5 The content is used to obtain a semantic grayscale image.

[0123] Alternatively, the electronic device can also acquire information such as... Figure 6 The reference image shown in (c) is used. Afterwards, the electronic device can... Figure 6 The three paintings in (a), (b), and (c) Figure 1 The input is fed into the image alignment module, and the result is as follows: Figure 6 The aligned semantic grayscale image shown in (d) is shown in the figure.

[0124] In some embodiments, when aligning the viewpoints of the semantic grayscale image and the reference image, either the angle of the semantic grayscale image or the angle of the reference image can be adjusted. Regardless of which image's angle is adjusted, the final semantic grayscale image should have the same angle as the reference image.

[0125] In some embodiments, if the angle of the reference image is adjusted, the semantic grayscale image is used as the standard, and then the angle of the reference image is adjusted according to the semantic grayscale image until the reference image is aligned with the semantic grayscale image, and the aligned reference image is obtained and output.

[0126] For example, such as Figure 6 As shown, Figure 6 (a) in the image is the initial image captured by the main camera. After inputting the initial image captured by the main camera into the semantic partitioning module in the partitioning module, the result is as follows: Figure 6 The semantic grayscale image shown in (b) can also be acquired by the electronic device through a secondary camera, such as... Figure 6 The reference image shown in (c) is used. Afterwards, the electronic device can... Figure 6 The three paintings in (a), (b), and (c) Figure 1 The input is sent to the image alignment module, and Figure 6 The angle of the reference image shown in (c) is adjusted to Figure 6 After adjusting the angle of the semantic grayscale image shown in (b), an aligned semantic grayscale reference image is obtained.

[0127] In some examples, the electronic device inputs a reference image captured by a secondary camera into a semantic partitioning module for auxiliary partitioning. Partitioning is performed based on the reference image and the initial image, with image alignment occurring during the partitioning process. Finally, an aligned semantic grayscale image is output. In this case, the aligned semantic grayscale image can be obtained directly, eliminating the need for an image alignment module in the image processing.

[0128] For example, such as Figure 7 As shown, Figure 7 (a) in the image is the initial image captured by the main camera. Figure 7 The reference image captured by the secondary camera, as shown in (b), is used to input the initial image captured by the main camera and the reference image captured by the secondary camera into the semantic partitioning module to obtain the following result: Figure 7 The aligned semantic grayscale image is shown in (c). Then, the aligned semantic grayscale image (c) is input into the dual-source estimation model to obtain the dual-source estimation result.

[0129] Optionally, both the initial image captured by the main camera and the reference image captured by the secondary camera are in RAW format.

[0130] Optionally, the image alignment module determines the spatial mapping relationship between the initial image captured by the main camera and the reference image captured by the secondary camera, and then maps the semantic grayscale image into the aligned semantic grayscale image corresponding to the reference image according to the spatial mapping relationship.

[0131] Optionally, the image alignment module may use methods including, but not limited to, combinations of image operators and feature matching methods, or methods based on deep learning model architectures (CNN, Transformer, etc.). Image operators may include, for example, scale-invariant feature transform (SIFT), speeded uprobust features (SURF), and oriented fast and rotated brief (ORB). Feature matching methods may include, for example, random sample consensus (RANSAC).

[0132] For example, feature point matching is performed on the initial image and the reference image using image operators. Then, the correspondence between each feature point in the initial image and the reference image is determined using a feature matching method, resulting in a mapping matrix of feature point correspondences. The semantic grayscale image is then adjusted based on this mapping matrix, changing the perspective of the semantic grayscale image to that of the reference image, resulting in an aligned semantic grayscale image.

[0133] For example, such as Figure 8 As shown, Figure 8 (a) in the image is the initial image. Figure 8 In the image (b), the reference image is used. Feature points in the initial image are determined as follows: 1 for hair, 2 for shoulder, 3 for finger, and 4 for clothing hem; feature points in the reference image are determined as follows: 5 for hair, 6 for shoulder, 7 for finger, and 8 for clothing hem. Then, the feature points of the initial and reference images are matched to obtain correspondences, such as 1 and 5, 2 and 6, 3 and 7, and 4 and 8. Based on the correspondence between the reference points of the initial and reference images, the coordinate changes of the initial image are determined. Then, the coordinates of each feature point in the semantic grayscale image are adjusted according to the coordinate changes of the initial image to obtain the aligned semantic grayscale image.

[0134] Optionally, if the electronic device inputs an initial image and a reference image to the partitioning module, image alignment is unnecessary, and the partitioning module directly outputs the aligned semantic grayscale image, improving partitioning efficiency. Instead of directly outputting the semantic grayscale image to the dual-white-point estimation model for dual-white-point determination, the electronic device uses a reference image with more channels as auxiliary information in the light source determination process. Even if the specific information of the semantic grayscale image cannot be determined, the light source information can be more accurately determined based on the reference image, leading to more accurate dual-white-point information. This is not limited by the accuracy of semantic recognition and allows for large-scale application.

[0135] In some embodiments, after the electronic device obtains the aligned semantic grayscale image, the viewing angle of the aligned semantic grayscale image is the same as that of the reference image. The light source information extracted from the reference image can be mapped onto the aligned semantic grayscale image. The aligned semantic grayscale image and the reference image are then input into a dual white point estimation model. The dual white point estimation model extracts different light source information from the reference image, then maps the light source information of the reference image to the aligned semantic grayscale image, and determines the corresponding dual white point information based on the aligned semantic grayscale image and the light source information. The dual white point information includes primary white point information and secondary white point information.

[0136] Optionally, the light source information includes not only the number of light sources, but also the coverage area of ​​each light source, i.e., the partitioning result of each light source. Furthermore, the direction, intensity, and other information of the light source can be determined based on the numerical information of each pixel contained in the light source information.

[0137] For example, such as Figure 6 As shown, the image alignment module inputs the aligned semantic grayscale image (d) and the reference image (c) into the dual white point estimation model (i.e., the dual light source estimation model). The dual white point estimation model extracts different light source information from the reference image (c) based on the multi-channel information of the reference image (c), then maps the light source information of the reference image (c) to the aligned semantic grayscale image (d), and determines the corresponding dual white point information based on the aligned semantic grayscale image (d) and the light source information, i.e., the dual light source estimation result.

[0138] Optionally, the number of white points is preset in the dual white point estimation model, typically 2. If the hardware of the electronic device supports it, the number of white points in the dual white point estimation model can be set to more according to the actual situation. This application embodiment takes the dual white point estimation model as an example, preset the number of white points in the image to include 2, and introduces the image processing method provided by this application embodiment.

[0139] It should be understood that the image processing method provided in the embodiments of this application can also be applied to image processing scenarios with a larger number of white points, and image processing can be performed through a multi-white-point estimation model.

[0140] Optionally, the number of white points is 2 or more provided that the reference image is a multi-source image.

[0141] Optionally, the more important subject region in the aligned semantic grayscale image can be identified as the main white point adjustment region, for example... Figure 6 The portrait area shown in (b) is the primary white point adjustment area. Secondary areas or background areas are designated as secondary white point adjustment areas, for example... Figure 6 The environmental area shown in (c) is the secondary white point adjustment area. The method for determining the corresponding adjustment areas of the primary and secondary white points can be flexibly adjusted according to the actual situation.

[0142] Optionally, the more important subject area in the aligned semantic grayscale image is not limited to the human figure area, but can also be the object area, scene area, etc.

[0143] Optionally, if there are two or more different light source features in the reference image, the number of white points is usually still set to double white points. The double white point estimation model determines the regions corresponding to the two main light sources in the reference image as the adjustment regions for double white points.

[0144] Optionally, the portions of grayscale values ​​in the aligned semantic grayscale image that are at their extreme values ​​can be designated as primary or secondary white points. For example, portions with a grayscale value of 1 can be designated as primary white points, and portions with a grayscale value of 0 can be designated as secondary white points. If the grayscale value is an intermediate value, the primary and secondary white points are linearly superimposed and then corrected.

[0145] Thus, the dual white point estimation model receives the aligned semantic grayscale image. Since the distinction between the person and the background is relatively clear in the aligned semantic grayscale image, the model separates the spatial features of the foreground person and the background according to the aligned semantic grayscale image during primary and secondary white point estimation. This eliminates the ambiguity of white points caused by different light sources in space, making the white point estimation results more accurate and stable, and solving the problems of color cast and flickering in the person and background areas in real-world scenes. This achieves more accurate color reproduction in photography scenarios and improves the accuracy of the algorithm results.

[0146] For example, such as Figure 4 As shown, the aligned partitioned grayscale image output by the partitioning module in the electronic device and the double white point estimation result output by the double white point estimation model are input to the image adjustment module. The image adjustment module performs color correction on the initial image based on the double white point information to obtain the final target image with correct color. The image adjustment module can be an image signal processor (ISP) active path, etc.

[0147] Optionally, each white point estimation result in the dual white point estimation results represents the color of white within the corresponding light source region in the aligned partitioned grayscale image. The image adjustment module determines the color of the target white point, which is the color that the adjusted white should ultimately present, typically standard white light, such as D65 simulated average daylight. The image adjustment module maps the light source region corresponding to each white point estimation result in the aligned partitioned grayscale image to the initial image. Then, the image adjustment module performs color correction on the corresponding light source region in the initial image based on each white point estimation result, obtaining the final target image with correct color. Specifically, color correction adjusts the color value of white corresponding to the white point estimation result within the light source region to the color value of white corresponding to the target white point.

[0148] Figure 9 This is a flowchart illustrating an image processing method provided in an embodiment of this application. It should be noted that this method does not rely on... Figure 9 The specific order described below is a limitation. It should be understood that in other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps: S901. The user-instructed shooting operation was detected, and the initial image and reference image were acquired.

[0149] Optionally, the electronic device includes at least one camera, wherein the initial image and the reference image are images captured by the same camera or images captured by different cameras.

[0150] Optionally, the initial image and the reference image can be the same image, or they can be different images. Where the initial image and the reference image are the same, it means that the initial image and the reference image are a single image. When the initial image and the reference image are different images, they can be images that differ in any of the following aspects: angle, image content, image quality, etc.

[0151] Optionally, if the initial image and the reference image are different images, the number of channels in the initial image is less than the number of channels in the reference image, and the resolution of the initial image is greater than the resolution of the reference image.

[0152] For example, if an electronic device includes a camera, the same initial image and reference image can be captured through the camera, or different initial images and reference images can be captured.

[0153] For example, if an electronic device includes multiple cameras, an initial image is captured by the main camera among the multiple cameras, and a reference image is captured by the secondary cameras among the multiple cameras. Although the images are captured by different cameras, the initial image and the reference image can be the same image, or the initial image and the reference image can be different images.

[0154] The following description uses an electronic device that includes two cameras, with the initial image and reference image being images captured by different cameras.

[0155] In some embodiments, the electronic device acquires an initial image in raw image file format via a first camera, and acquires a reference image corresponding to the initial image via a second camera.

[0156] Optionally, RAW format is the raw data set obtained after the image sensor converts the captured light signals into digital signals, without any processing. It is a data packet that records the most original and complete light information of the shooting scene. RAW format files record the intensity (i.e., brightness) of light perceived by each pixel in the initial image, rather than direct color information.

[0157] Optionally, the initial image has spatial resolution, which refers to the level of detail that can be distinguished in the image.

[0158] Optionally, the reference dimensions for spatial resolution are (H, W, C), where H > 1 and W > 1. Here, H is Height, representing the number of pixels in the vertical direction of the initial image; W is Width, representing the number of pixels in the horizontal direction of the initial image; and C is Channels, describing the number of information components required for each pixel. For example, common color images typically have three channels: red, green, and blue. The colors from these three channels are mixed together to determine the final color of the pixel. Grayscale images have only one channel, storing only brightness information. Portable network graphic format (PNG) images with transparency have four channels: red, green, blue, and alpha.

[0159] In some embodiments, the second camera is a camera that acquires multi-channel information. The reference image obtained based on the multi-channel information can include more accurate spectral information. Therefore, the electronic device can more accurately determine the number of light sources corresponding to the reference image based on the spectral information.

[0160] Optionally, the initial image captured by the first camera and the reference image captured by the second camera are images with different numbers of channels but identical or similar content. Therefore, the number of light sources in the reference image is also the number of light sources in the initial image. The electronic device can then determine the number of light sources in the reference image using a preset method, and thus determine the number of light sources in the initial image.

[0161] Optionally, the preset methods may include determining the light source data based on the number of cluster centers, determining the number of light sources based on color deviation, etc.

[0162] In some examples, the electronic device projects the color values ​​(i.e., spectral information) of all pixels in a reference image into a high-dimensional color space, and then uses a clustering algorithm to cluster the pixel color values, with the cluster centers considered as potential light sources. Therefore, the number of cluster centers is determined as the number of light sources. Each cluster center represents the color distribution under a dominant light source.

[0163] Optionally, the electronic device acquires more new feature information corresponding to the color value, and determines the information composed of the color value and the new feature information as the information corresponding to the high-dimensional color space. The new feature information is discriminative, providing additional information that makes the originally mixed data separable, such as transparency, chroma, brightness, and saturation. Chroma includes, for example, the red-to-green ratio (R / G) and the red-to-blue ratio (R / B). If some pixels exhibit similar color values ​​in the spectral information, they are distinguished using the information corresponding to the high-dimensional color space. These pixels with similar color values ​​show significant differences in the new feature information, allowing for more accurate differentiation of the light source information of these pixels.

[0164] In other examples, the electronic device directly determines the reference image captured by the second camera as an image with a light source count of 1, and then performs global correction on the reference image to obtain a corrected reference image. The electronic device then determines whether color cast still exists in the corrected reference image. If color cast still exists in the corrected reference image, it determines that the number of light sources in the reference image captured by the second camera is greater than 1.

[0165] Optionally, color cast refers to an image's colors deviating from their proper, true, or expected colors, presenting an unnatural color bias.

[0166] In some embodiments, if the number of light sources in the initial image is 1, the electronic device determines that the initial image is a single-light source image. When performing color correction on the initial image, there is no need to partition the light sources; a global adjustment can be performed directly. If the number of light sources in the initial image is greater than 1, the electronic device determines that the initial image is a multi-light source image. When color correction is required on the initial image, the light source information in the initial image must first be determined, and the initial image must be partitioned by light sources.

[0167] Optionally, if the number of light sources in the initial image is greater than 1, the electronic device will generally assume that the number of light sources in the initial image is 2.

[0168] Optionally, in some complex scenarios, the number of light sources can be set to 3 or more.

[0169] This application uses an example with two light sources to illustrate the image processing method provided in this application.

[0170] S902. Based on the initial image, obtain the first image and the second image.

[0171] The first image includes the semantic partitioning result of the initial image, and the second image includes the light source partitioning result of the initial image.

[0172] In some embodiments, based on the offset angle information of the reference image, a first image aligned with the reference image is obtained according to the initial image.

[0173] In some embodiments, after determining the number of light sources, if the number of light sources is greater than 1, the initial image is determined to be a multi-light source scene. This can easily lead to partitioning errors during the light source partitioning process. Therefore, a semantic partitioning module is added to the partitioning module of the electronic device to assist in identifying semantic information in the initial image, so as to accurately determine the light source information corresponding to the semantic information.

[0174] In some embodiments, the partitioning module in the electronic device inputs the initial image to the semantic partitioning module of the color correction model for semantic feature extraction, resulting in a semantic grayscale image (mask) containing semantic information from the initial image, i.e., the first image. For example, when the semantic partitioning model extracts portrait features, it obtains a first image (mask) containing portrait information from the initial image. The semantic partitioning module can extract arbitrary semantic features during feature extraction, such as portrait features, object features, scene features, etc.

[0175] Optionally, the semantic partitioning module identifies the human image semantic information in the initial image, extracts the pixel information identified as human image semantic information, and obtains the corresponding first image containing the human image information in the initial image.

[0176] Optionally, after the initial image is input into the semantic partitioning module in the electronic device, the partitioning module determines the semantic information to be identified according to different recognition requirements, such as the semantic information being a human figure or an object.

[0177] Optionally, the feature extraction methods set in the semantic partitioning module and the light source partitioning module of the electronic device include, but are not limited to, methods based on deep learning model architectures. These deep learning model architecture-based methods include, for example, convolutional neural networks (CNNs) and self-attention mechanisms (transformers).

[0178] Thus, the first image containing the human image information from the initial image is the output of the semantic partitioning module. Since the semantic partitioning module can acquire a large amount of training data through various means such as open-source data and ordinary cameras, the confidence of its output is improved after training with the acquired training data. Therefore, the semantic partitioning module can achieve accurate and stable output results, resolving issues such as ambiguity in the light source of the human image and flickering between video frames.

[0179] In some embodiments, the partitioning module in the electronic device inputs the initial image to the light source partitioning module of the color correction model. The light source partitioning module identifies the light source feature information in the initial image, outputs the gray value of the light source feature information corresponding to each pixel in the initial image, and then partitions the light source according to the output gray value to obtain a second image (mask) containing the light source information in the initial image.

[0180] Optionally, the light source feature information in the initial image can be identified. For example, pixels with similar color temperatures in the initial image can be divided into the same region, light sources can be partitioned according to different objects or regions (such as sky, ground, etc.) in the initial image identified semantically, or light sources can be partitioned according to regions manually selected by the user.

[0181] For example, such as Figure 5 As shown, the electronic device acquires an initial image (a) in the original image file format through the first camera. Then, the electronic device can input the initial image (a) into the partitioning module. The semantic partitioning module in the partitioning module performs feature extraction on the initial image (a) and identifies the 01 region in the initial image (a) as human portrait information. Therefore, the 01 region in the initial image (a) is extracted to obtain the corresponding first image (b) containing the human portrait information in the initial image.

[0182] Optionally, the electronic device can also extract features from the initial image (a) through the light source partitioning module in the partitioning module, identify that region 02 in the initial image (a) is the ground, regions 03 and 04 are the wall, region 05 is the indoor ceiling wall, and region 06 is the door, extract the light intensity of regions 02 to 06, and generate a second image (c) with corresponding depth according to the different light intensity of each region.

[0183] Optionally, during the acquisition of the first image (b), the portrait is typically represented by a uniform color, such as black. In some cases, the first image can be generated with a color gradient based on the lighting conditions of the portrait portion of the actual initial image.

[0184] Optionally, the second image (c) output by the light source partitioning module is an image carrying pixel information of different brightness, that is, it can represent the transitional changes in the brightness of light in the initial image.

[0185] Optionally, the second image is a grayscale image. A grayscale image (mask) is a grayscale image with the same size as the initial image, used to selectively cover or display certain areas of the initial image, thereby enabling precise manipulation of specific areas.

[0186] In this way, the light source information in the initial image can be accurately identified based on the second image. By introducing semantic information for partitioning, subsequent image processing operations (such as white balance adjustment) are no longer global adjustments, but can be precisely and locally adjusted, thereby achieving local correction for complex mixed light source scenes. Even when the semantic information in the initial image is uncertain, the semantic information can be protected by separate partitioning, avoiding incorrect color processing of the semantic information and improving generalization.

[0187] S903. Obtain the third image based on the first and second images.

[0188] In some embodiments, the partitioning module merges the first image output by the semantic partitioning with the second image output by the light source partitioning module set in the partitioning module to obtain the final accurate partitioning result of the initial image, that is, the third image.

[0189] In some embodiments, the second image and the first image are input to the fusion module of the color correction model of the electronic device, and the second image and the first image are fused by a preset fusion method in the fusion module to obtain a third image.

[0190] For example, such as Figure 5 As shown, the third image (d) finally output by the partitioning module of the electronic device contains the first image (b) and the second image (c). It can accurately display the area of ​​light source information in the initial image and completely and accurately divide the area corresponding to the semantic information, which is beneficial to subsequent local white balance processing.

[0191] Optionally, the preset fusion methods include, but are not limited to, image domain filtering algorithms and methods based on deep learning model architectures. Image domain filtering algorithms include bilateral filtering algorithms, guided filtering algorithms, bilateral optimizer algorithms, etc. Methods based on deep learning model architectures include convolutional neural networks, self-attention mechanisms, etc.

[0192] Thus, since the output of the semantic partitioning module is relatively stable and has high accuracy and robustness, the accuracy of fusing the grayscale images output by the semantic partitioning module and the light source partitioning module to obtain the third image is improved, and the problem of light source ambiguity is solved.

[0193] In some embodiments, since the confidence level of the first image is higher than that of the second image, the third image obtained after fusing the first and second images achieves accurate and stable semantic information segmentation, solving problems such as semantic information light source ambiguity and video frame flickering. Because a more accurately segmented third image is obtained, different segments can be selected for adjustment more accurately when performing image color adjustment based on the third image.

[0194] S904. Obtain the light source estimation result based on the reference image and the first image.

[0195] The light source estimation result (i.e., the white point estimation result) indicates information about at least two light sources (i.e., white points) included in the reference image.

[0196] In some embodiments, when there is a relative angular deviation between the initial image and the reference image, the light source estimation result is obtained after aligning the reference image and the first image according to the relative angular deviation.

[0197] In some embodiments, during the dual-light source estimation process, because the reference image has a large number of channels, the electronic device can more accurately determine the light source based on the number of channels in the reference image, thus obtaining a more accurate dual-light source estimation result. The dual-light source estimation module maps the light source information of the reference image onto the first image to determine the dual-light source estimation result. If there is a viewing angle deviation between the reference image and the initial image, it indicates that there is also a viewing angle deviation between the first image and the reference image, and the light source information of the reference image cannot be accurately mapped.

[0198] In some embodiments, the first image is aligned with the reference image based on the relative angle deviation; and the light source estimation result is obtained based on the aligned first image and the reference image.

[0199] Optionally, before performing dual-source estimation, the viewing angles of the first image and the reference image need to be aligned. The first image, the reference image captured by the second camera, and the initial image captured by the first camera are input into the image alignment module of the color correction model of the electronic device to adjust the first image. Using the reference image as a standard, the angle of the first image is adjusted according to the reference image until the angles of the first image and the reference image are aligned, resulting in the aligned first image, which is then output.

[0200] Optionally, since the initial image and the reference image capture different content areas, performing alignment operations in this case may result in the image angles and content not being perfectly aligned. During the alignment process, the initial image and the reference image can be adjusted until the overlap is greater than or equal to a preset overlap threshold before output, without requiring perfect alignment. The overlap threshold can be a custom value such as 70% or 80%.

[0201] For example, such as Figure 6 As shown, Figure 6 In the diagram, (a) represents the initial image captured by the first camera. After inputting the initial image captured by the first camera into the semantic partitioning module of the partitioning module, the result is as follows: Figure 6 The first image shown in (b) above. For example, refer to the above. Figure 5 The content is described above, and the first image is obtained.

[0202] Optionally, the electronic device can also acquire information such as through a second camera. Figure 6 The reference image shown in (c) is used. Afterwards, the electronic device can... Figure 6 The three paintings in (a), (b), and (c) Figure 1 The input is fed into the image alignment module, and the result is as follows: Figure 6 The first image after alignment is shown in (d) in the image.

[0203] In some embodiments, the reference image is aligned with the initial image based on the relative angle deviation; and the light source estimation result is obtained based on the aligned reference image and the first image.

[0204] Optionally, the initial image can be determined as a standard image, and then the reference image can be adjusted according to the angle of the initial image until the angle of the reference image is aligned with that of the initial image, thus obtaining the aligned reference image and outputting it.

[0205] Optionally, there may be differences in resolution and number of channels between the reference image and the initial image. The initial image acquired by the first camera is typically a three-channel image, but in some special cases it may be an image with a different number of channels. Generally, the reference image acquired by the second camera has a greater number of channels than the initial image acquired by the first camera, and the resolution of the reference image acquired by the second camera is lower than the resolution of the initial image acquired by the first camera.

[0206] In some embodiments, light source information corresponding to a reference image is obtained; and a light source estimation result is obtained based on the first image and the light source information.

[0207] In some examples, the electronic device inputs a reference image captured by a second camera into a semantic partitioning module for auxiliary partitioning. Partitioning is performed based on the reference image and the initial image, with image alignment occurring during the partitioning process. Finally, an aligned first image is output. In this case, the aligned first image can be obtained directly, eliminating the need for an image alignment module in the image processing.

[0208] For example, such as Figure 7 As shown, Figure 7 (a) in the image is the initial image captured by the first camera. Figure 7 The reference image captured by the second camera, as shown in (b), is used to input the initial image captured by the first camera and the reference image captured by the second camera into the semantic partitioning module to obtain the following result: Figure 7 The first image after alignment is shown in (c) of the image.

[0209] Optionally, the initial image captured by the first camera and the reference image captured by the second camera are both in RAW format.

[0210] Optionally, the image alignment module determines the spatial mapping relationship between the initial image captured by the first camera and the reference image captured by the second camera, and then maps the first image into the aligned first image corresponding to the reference image according to the spatial mapping relationship.

[0211] Optionally, the image alignment module may use methods including, but not limited to, combinations of image operators and feature matching methods, or methods based on deep learning model architectures. Image operators may include, for example, scale-invariant feature transformations, accelerated robust features, and fast feature point extraction and description algorithms. Feature matching methods may include, for example, random sample consensus algorithms.

[0212] For example, feature point matching is performed on the initial image and the reference image using image operators. Then, the correspondence between each feature point in the initial image and the reference image is determined using a feature matching method, resulting in a mapping matrix of feature point correspondences. The first image is then adjusted according to this mapping matrix, changing its viewpoint to that of the reference image, resulting in an aligned first image.

[0213] For example, such as Figure 8 As shown, Figure 8 (a) in the image is the initial image. Figure 8 In the first image, (b) is the reference image. Feature points in the initial image are determined as follows: 1 for hair, 2 for shoulders, 3 for fingers, and 4 for clothing hem; feature points in the reference image are determined as follows: 5 for hair, 6 for shoulders, 7 for fingers, and 8 for clothing hem. Then, the feature points of the initial image and the reference image are matched to obtain correspondences, such as 1 and 5, 2 and 6, 3 and 7, and 4 and 8. Based on the correspondence between the reference points of the initial and reference images, the coordinate changes of the initial image are determined. Then, the coordinates of each feature point in the first image are adjusted according to the coordinate changes of the initial image to obtain the aligned first image.

[0214] Thus, if the electronic device inputs an initial image and a reference image to the partitioning module, image alignment is unnecessary; the partitioning module directly outputs the aligned first image, improving partitioning efficiency. Instead of directly outputting the first image to the dual-light source estimation model for dual-light source determination, the electronic device uses a reference image with more channels as auxiliary information in the light source determination process. Even if the specific information of the first image cannot be determined, the light source information can be more accurately determined based on the reference image, leading to more accurate dual-light source information. This approach is not limited by the accuracy of semantic recognition and can be applied on a large scale.

[0215] In some embodiments, after the electronic device obtains the aligned first image, the viewing angle of the aligned first image is the same as that of the reference image. The light source information extracted from the reference image can be mapped onto the aligned first image. The aligned first image and the reference image are then input into a dual-light source estimation model. The dual-light source estimation model extracts different light source information from the reference image, then maps the light source information of the reference image to the aligned first image, and determines the corresponding dual-light source information based on the aligned first image and the light source information. The dual-light source information includes primary light source information and secondary light source information.

[0216] Optionally, the light source information includes not only the number of light sources, but also the coverage area of ​​each light source, i.e., the partitioning result of each light source. Furthermore, the direction, intensity, and other information of the light source can be determined based on the numerical information of each pixel contained in the light source information.

[0217] For example, such as Figure 6 As shown, the image alignment module inputs the aligned first image (d) and the reference image (c) into the dual-source estimation model. The dual-source estimation model extracts different source information from the reference image (c) based on the multi-channel information of the reference image (c), then maps the source information of the reference image (c) to the aligned first image (d), and determines the corresponding dual-source information based on the aligned first image (d) and the source information.

[0218] In some embodiments, the light source estimation result of the target number is obtained based on the first image and light source information.

[0219] Optionally, the number of light sources is preset in the dual-source estimation model. Typically, the number of light sources is 2. If the hardware of the electronic device supports it, the number of light sources in the dual-source estimation model can be set according to the actual situation.

[0220] This application takes a dual-source estimation model as an example, with a preset image containing two source sources, and introduces the image processing method provided in this application embodiment.

[0221] It should be understood that the image processing method provided in the embodiments of this application can also be applied to image processing scenarios with a larger number of light sources, and image processing can be performed through a multi-light source estimation model.

[0222] Optionally, the number of light sources is two or more provided that the reference image is a multi-light source image.

[0223] Optionally, the more important subject area in the aligned first image can be identified as the main light source adjustment area, for example... Figure 6 The portrait area shown in (b) is the primary light source adjustment area. Secondary areas or background areas are designated as secondary light source adjustment areas, for example... Figure 6The environmental area shown in (c) is the secondary light source adjustment area. The method for determining the corresponding adjustment areas of the primary and secondary light sources can be flexibly adjusted according to the actual situation.

[0224] Optionally, the more important subject area in the aligned first image is not limited to the portrait area, but can also be the object area, scene area, etc.

[0225] Optionally, if there are two or more different light source feature information in the reference image, the number of light sources is usually still set to two light sources. The two light source estimation model determines the regions corresponding to the two main light sources in the reference image as the adjustment regions of the two light sources.

[0226] Optionally, the regions with extreme grayscale values ​​in the aligned first image can be designated as either primary or secondary light sources. For example, regions with a grayscale value of 1 can be designated as primary light sources, and regions with a grayscale value of 0 as secondary light sources. If the grayscale value is an intermediate value, the primary and secondary light sources are linearly superimposed and then corrected.

[0227] Thus, the dual-light source estimation model receives the aligned first image. Since the distinction between the person and the background is relatively clear in the aligned first image, the spatial features of the foreground person and the background are processed separately during the estimation of primary and secondary light sources. This eliminates the ambiguity of light sources caused by different light sources in space, making the light source estimation results more accurate and stable. The color of the initial image is optimized based on the reference image, solving the color cast and flickering problems in the person and background areas in real-world scenes. This achieves more accurate color reproduction in photography scenarios and improves the accuracy of the algorithm results.

[0228] S905. Based on the third image and the light source estimation results, adjust the color of the initial image to obtain the target image.

[0229] For example, such as Figure 4 As shown, the aligned third image output by the partitioning module in the electronic device and the dual-light source estimation result output by the dual-light source estimation model are input to the image adjustment module. The image adjustment module performs color correction on the initial image based on the dual-light source information to obtain the final target image with correct color. The image adjustment module can be an image signal processor (ISP) active path, etc.

[0230] Optionally, each light source estimation result in the dual-light source estimation results represents the color of white within the corresponding light source region in the aligned third image. The image adjustment module determines the color of the target light source, which is the color that white should ultimately appear after adjustment, typically standard white light, such as D65 simulated average daylight. The image adjustment module maps the light source region corresponding to each light source estimation result in the aligned third image to the initial image. Then, the image adjustment module performs color correction on the corresponding light source region in the initial image based on each light source estimation result to obtain the final target image with correct color. Specifically, color correction adjusts the color value of white corresponding to the light source estimation result within the light source region to the color value of white corresponding to the target light source.

[0231] S906. Display the target image.

[0232] The above combination Figures 4-9 The image processing method provided in the embodiments of this application is described in detail below. Figure 10 This application provides a detailed description of the electronic device 1000 provided in the embodiments.

[0233] In one possible design, Figure 10 This is a schematic diagram of the structure of the electronic device 1000 provided in an embodiment of this application. Figure 10 As shown, the electronic device 1000 may include a transceiver unit 1001 and a processing unit 1002. The electronic device 1000 can be used to implement the functions of the electronic device involved in the above method embodiments.

[0234] Optionally, the transceiver unit 1001 is used to support the electronic device 1000 in performing... Figure 9 S901 in the middle.

[0235] Optionally, the processing unit 1002 is used to support the electronic device 1000 in performing operations. Figure 9 S902 to S905 in the series.

[0236] The transceiver unit may include a receiving unit and a transmitting unit, and may be implemented by a transceiver or transceiver-related circuit components, and may be a transceiver or transceiver module. The operation and / or function of each unit in the electronic device 1000 are respectively to implement the corresponding flow of the image processing method described in the above method embodiments. All relevant content of each step involved in the above method embodiments can be referred to the functional description of the corresponding functional unit, and for the sake of brevity, it will not be repeated here.

[0237] Optionally, Figure 10 The electronic device 1000 shown may also include a storage unit ( Figure 10(Not shown in the image), this storage unit stores a program or instruction. After the transceiver unit 1001 and the processing unit 1002 execute the program or instruction, it causes... Figure 10 The electronic device 1000 shown can perform the image processing method described in the above method embodiments.

[0238] Figure 10 The technical effects of the electronic device 1000 shown can be referred to the technical effects of the image processing method described in the above method embodiments, and will not be repeated here. In addition to being in the form of electronic device 1000, the technical solution provided in this application can also be a functional unit or chip in electronic device 1000, or a device used in conjunction with electronic device 1000.

[0239] This application also provides a chip system, including: a processor coupled to a memory, the memory being used to store programs or instructions, wherein when the program or instructions are executed by the processor, the chip system implements the methods in any of the above method embodiments.

[0240] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0241] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0242] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0243] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0244] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, it causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0245] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0246] In addition, this application also provides an apparatus. Specifically, the apparatus may be a component or module, and may include one or more processors and a memory connected together. The memory stores a computer program. When the computer program is executed by one or more processors, the apparatus performs the image processing methods described in the above-described method embodiments.

[0247] The apparatus, computer-readable storage medium, computer program product, or chip provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0248] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC).

[0249] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the above functional modules is only used as an example. In practical applications, the above functions can be assigned to different functional modules as needed; that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0250] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of modules or units may be electrical, mechanical or other forms.

[0251] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0252] Computer-readable storage media include, but are not limited to, any of the following: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media capable of storing program code.

[0253] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, the method includes: Upon detecting a user-instructed action to take a picture, the system acquires an initial image and a reference image. Based on the initial image, a first image is obtained; wherein the first image includes the semantic partitioning result of the initial image; Based on the reference image and the first image, obtain the light source estimation result; Based on the first image and the light source estimation result, the color of the initial image is adjusted to obtain the target image; Display the target image.

2. The method according to claim 1, characterized in that, The first image also includes the light source partitioning results of the initial image.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the first image based on the initial image includes: Based on the initial image, partitioning results are obtained; wherein, the partitioning results include semantic partitioning results and / or light source partitioning results; Based on the partitioning results, the first image is obtained.

4. The method according to claim 3, characterized in that, The step of obtaining the partitioning result based on the initial image includes: Based on the initial image, the semantic partitioning result is obtained; or The semantic partitioning result is obtained based on the initial image and the reference image.

5. The method according to claim 3 or 4, characterized in that, The step of obtaining the partitioning result based on the initial image includes: Based on the initial image, obtain the light source partitioning result; or The light source partitioning result is obtained based on the initial image and the reference image.

6. The method according to any one of claims 1 to 5, characterized in that, The step of obtaining the light source estimation result based on the reference image and the first image includes: In the event of a relative angular deviation between the initial image and the reference image, the light source estimation result is obtained after aligning the reference image and the first image according to the relative angular deviation.

7. The method according to claim 6, characterized in that, After aligning the reference image and the first image according to the relative angle deviation, obtaining the light source estimation result includes: Align the first image with the reference image based on the relative angle deviation; The light source estimation result is obtained based on the aligned first image and the reference image.

8. The method according to claim 6, characterized in that, After aligning the reference image and the first image according to the relative angle deviation, obtaining the light source estimation result includes: Align the reference image with the initial image based on the relative angle deviation; The light source estimation result is obtained based on the aligned reference image and the first image.

9. The method according to claim 1, characterized in that, The step of obtaining the first image based on the initial image includes: Based on the offset angle information of the reference image, the first image aligned with the reference image is obtained according to the initial image.

10. The method according to any one of claims 1 to 9, characterized in that, The step of obtaining the light source estimation result based on the reference image and the first image includes: Obtain the light source information corresponding to the reference image; Based on the first image and the light source information, obtain the light source estimation result.

11. The method according to claim 10, characterized in that, The step of obtaining the light source estimation result based on the first image and the light source information includes: Based on the first image and the light source information, the light source estimation result of the target number is obtained.

12. The method according to any one of claims 1 to 11, characterized in that, The light source estimation result indicates information about at least two light sources included in the reference image.

13. The method according to any one of claims 1 to 12, characterized in that, The electronic device includes at least one camera, and the initial image and the reference image are images captured by the same camera or images captured by different cameras.

14. The method according to any one of claims 1 to 13, characterized in that, The initial image is the same as the reference image, or the initial image is different from the reference image.

15. The method according to claim 14, characterized in that, When the initial image and the reference image are different images, the number of channels in the initial image is less than the number of channels in the reference image.

16. The method according to claim 14, characterized in that, In the case where the initial image and the reference image are different images, the resolution of the initial image is greater than the resolution of the reference image.

17. An electronic device, characterized in that, include: A processor and a memory, the processor including a plurality of graphics processors, the memory coupled to the processor, the memory for storing computer program code including computer instructions, wherein when the processor reads the computer instructions from the memory, the electronic device causes the electronic device to perform the method as described in any one of claims 1-16.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-16.

19. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-16.

Images