Image processing method based on color adaptation, electronic equipment and related medium

Abstract

The embodiment of the invention provides an image processing method based on color adaptation, electronic equipment and a related medium, and the method comprises the steps: obtaining a RAW image in response to a photographing instruction of a user; performing local white balance processing on the RAW image to generate a first processing image corresponding to the RAW image; cutting the first processing image into i image blocks; performing color adaptation processing on each image block in the i image blocks in parallel to generate a second processing image corresponding to each image block; and splicing the second processing images corresponding to the image blocks to generate a target image. Through the implementation of the method, white balance processing can be performed on the RAW image, the first processing image is generated, then the first processing image is cut into a plurality of image blocks, and color adaptation processing is performed on the image blocks, so that full-pixel-level color adaptation processing can be performed on the first processing image on the premise that the image color adaptation processing rate is not affected, and the image processing efficiency is improved. And the image color adaptation effect is more obvious, so that the use experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to an image processing method based on color adaptation, an electronic device and related media. Background Art

[0002] With the enrichment of the camera functions of electronic devices, the scenes that users use electronic devices to shoot have also become more diverse, which has also led to users paying more and more attention to the camera effects of electronic devices. The camera effect of electronic devices is partly related to the model of the camera, and the other part is related to the image processing capability of the electronic device. In order to provide users with images that are closer to the observation effect of the human eye, the existing technology can perform color adaptation processing on the image based on the difference between the ambient color temperature value and the screen color temperature value, but this color adaptation processing method cannot be applied to images obtained in mixed light source scenes. Because the light source conditions of mixed light source scenes are relatively complex, if the image is color adapted based only on the difference between the ambient color temperature value and the screen color temperature value, it is easy to cause inaccurate colors of some objects in the image (for example, some objects in high and low color temperature mixed light source scenes are blue), thereby reducing the user's shooting experience. Therefore, how to improve the image color adaptation effect is a problem that technicians in this field urgently need to solve. Summary of the invention

[0003] In a first aspect, the present application provides an image processing method based on chromatic adaptation, an electronic device, and related media. The method may include:

[0004] Responding to a shooting instruction from a user, acquiring a RAW image;

[0005] Performing white balance processing on the RAW image to generate a first processed image corresponding to the RAW image;

[0006] Cut the first processed image into i image blocks, i ≥ 2, i is a positive integer;

[0007] Performing color adaptation processing on each of the i image blocks simultaneously in a multi-process form to generate a second processed image corresponding to each image block;

[0008] The second processed images corresponding to the image blocks are spliced ​​to generate a target image.

[0009] By implementing the method provided in the first aspect, the electronic device can obtain a RAW image in response to the user's shooting instruction, and perform white balance processing on the RAW image, thereby generating a first processed image, which helps to balance the color temperature and brightness of the image, etc., and prevent the image presented to the user from being partially overexposed or having a serious local color cast (such as bluish or yellowish), which helps to ensure the user's shooting experience. Further, the electronic device can also cut the first processed image into multiple image blocks, and perform color adaptation processing on each image block in parallel in the form of multiple processes to obtain multiple second processed images. It can be seen that the electronic device can take corresponding color adaptation processing based on the corresponding light conditions of each part (or each pixel) of the first processed image, which helps to improve the effect of image color adaptation processing, thereby providing the user with an image with better picture color, so that the user has a better shooting experience. However, the prior art does not take into account the distribution of light sources in the shooting scene, nor the complexity of the light received by each object in the image, and only performs color adaptation processing on the image based on the difference between the ambient color temperature value and the screen color temperature value, which easily leads to the problem of inaccurate color of objects in the image after color adaptation processing, and cannot provide users with a good shooting experience. Different from the prior art, the method of the present application fully considers the light conditions of each object in the image, and can perform color adaptation processing on each image block of the first processed image, which helps to ensure the color adaptation effect of each image block, thereby ensuring the imaging effect of each object in the image. Moreover, the method of the embodiment of the present application can adopt the form of multiple processes, and perform color adaptation processing on each image block at the same time, which helps to improve the image color adaptation processing effect while ensuring the rate of color adaptation processing, thereby further improving the user's shooting experience.

[0010] In some embodiments of the method provided in the first aspect, performing color adaptation processing on each image block in the i image blocks to generate a second processed image corresponding to each image block may include:

[0011] Based on the color information of the pixels contained in each image block, a chromatic adaptation gain matrix corresponding to each image block is determined, where the color information may include a color temperature value, a brightness coefficient, and a chromaticity value;

[0012] Based on the chromatic adaptation gain matrix corresponding to each pixel block, a second processed image corresponding to each image block is generated.

[0013] By implementing the method provided in the above embodiment, the electronic device can determine the chromatic adaptation gain matrix based on the color information of the pixels contained in each image block, so as to generate the second processed image corresponding to each image block based on each image block and its corresponding chromatic adaptation matrix, thereby achieving the purpose of performing chromatic adaptation processing on each image block. The color information of the pixel may include the color temperature value of the pixel, the brightness coefficient of the pixel, and the chromaticity value of the pixel.

[0014] In some embodiments of the method provided in the first aspect, performing white balance processing on the RAW image to generate a first processed image corresponding to the RAW image may include:

[0015] The RAW image is input into the local white balance model and the first processed image is output.

[0016] By implementing the method provided in the above embodiment, the present application can use a local white balance model to perform white balance processing on the RAW image to achieve the purpose of balancing the color temperature and brightness of the image, which helps to prevent the image presented to the user from being partially overexposed or having a serious local color cast (such as bluish or yellowish), thereby ensuring the user's shooting experience. The local white balance model is a neural network model.

[0017] In some embodiments of the method provided in the first aspect, the local white balance model is a model obtained by training using a mixed light source image as a training sample and a standard output image corresponding to the mixed light source image as a label;

[0018] There are n light sources in the shooting scene corresponding to the mixed light source image, n≥2, and n is a positive integer;

[0019] The standard output image corresponding to the mixed light source image is generated based on the mixed light source image and the target chromaticity value.

[0020] By implementing the method provided in the above embodiment, the local white balance model in the present application uses mixed light source images as training samples, which helps the local white balance model to cover more image shooting scenes, ensures that images collected or shot in various complex light source scenes can have good color balancing capabilities, and lays the foundation for subsequent color adaptation processing, thereby ensuring the user's shooting experience. Furthermore, the method of the embodiment of the present application can generate training labels for the local white balance model based on the mixed light source and the target chromaticity value, and through self-made training data pairs, it helps to ensure the iterative effect of the local white balance model and improve the color balancing ability of the local white balance model.

[0021] Implementing the method provided in the first aspect, in some embodiments, the target chromaticity value is generated based on the chromaticity value corresponding to each calibration image in n calibration images and the fusion ratio corresponding to each calibration image, and there is a one-to-one correspondence between the n calibration images and the n light sources.

[0022] By implementing the method provided in the above embodiment, the mixed light source image in the present application corresponds to n light sources and also corresponds to n calibration images. Further, it can be considered that there is a one-to-one correspondence between the n calibration images and the n light sources. The method of the embodiment of the present application can determine the target chromaticity value based on the chromaticity value corresponding to each calibration image in the n calibration images and the fusion ratio corresponding to each calibration image. By determining the target chromaticity value, it is helpful to generate a suitable standard output image for the mixed light source image, thereby ensuring the training effect of the local white balance model.

[0023] In some embodiments of the method provided in the first aspect, any one of the n calibration images is generated by preprocessing a single light source image corresponding to any one of the calibration images, and the preprocessing may include lens shading correction processing, image format conversion, and white balance calibration processing;

[0024] The shooting scene corresponding to the single light source image corresponding to any one of the calibration images has a light source corresponding to the any one of the calibration images.

[0025] Implementing the method provided in the above embodiment, the mixed light source image in the present application corresponds to n single light source images, and there is a one-to-one correspondence between the n single light source images and the above n light sources. After the single light source image has been processed by lens shading correction, image format conversion and white balance calibration, a corresponding calibration image can be generated. Among them, the lens shading correction process can correct the uneven image brightness distribution problem caused by the optical characteristics of the lens and the image sensor, so that the corrected image has the same brightness level in the entire field of view; the image format conversion mainly refers to converting the RAW image into a three-channel PNG image, so that the two G channel values ​​in the RAW image are merged into one G channel value, which is helpful for the subsequent calculation of the target chromaticity value; the white balance calibration process is to further balance the color of the image, reduce the color cast of the image, and make the color of each object in the image more in line with the actual situation. The above preprocessing helps to generate a suitable standard output image in the future, thereby ensuring the training effect of the local white balance model.

[0026] Implementing the method provided by the first aspect, in some embodiments, the calibration image is a PNG image;

[0027] The fusion ratio corresponding to each calibration image is generated based on the G channel value of each calibration image.

[0028] By implementing the method provided in the above embodiment, the calibration image in the present application is a three-channel PNG image. The PNG image can merge the two G channel values ​​in the RAW image into one G channel value, which is helpful for the subsequent calculation of the target chromaticity value, thereby achieving the purpose of producing a standard output image corresponding to the mixed light source image, thereby ensuring the training effect of the local white balance model.

[0029] In a second aspect, an embodiment of the present application provides an electronic device, which may include: an interaction module, an image acquisition module, and an image processing module;

[0030] An interactive module, which can be used to receive shooting instructions from users;

[0031] The image acquisition module can be used to obtain a RAW image in response to a user's shooting instruction;

[0032] The image processing module can be used to perform white balance processing on the RAW image to generate a first processed image corresponding to the RAW image;

[0033] The image processing module may also be used to cut the first processed image into i image blocks, where i≥2 and i is a positive integer;

[0034] The image processing module may also be used to perform color adaptation processing on each of the i image blocks in a multi-process form to generate a second processed image corresponding to each image block;

[0035] The image processing module can also be used to splice the second processed images corresponding to the various image blocks to generate a target image.

[0036] To implement the method provided in the second aspect, in some embodiments, the electronic device may further include:

[0037] The image processing module may also be used to determine the chromatic adaptation gain matrix corresponding to each image block based on the color information of the pixels contained in each image block, where the color information may include a color temperature value, a brightness coefficient, and a chromaticity value;

[0038] The image processing module can also be used to generate a second processed image corresponding to each image block based on the chromatic adaptation gain matrix corresponding to each pixel block.

[0039] To implement the method provided in the second aspect, in some embodiments, the electronic device may further include:

[0040] The image processing module can also be used to input the RAW image into the local white balance model and output a first processed image.

[0041] In some embodiments of the method provided in the second aspect, the local white balance model is a model obtained by training using a mixed light source image as a training sample and a standard output image corresponding to the mixed light source image as a label;

[0042] There are n light sources in the shooting scene corresponding to the mixed light source image, n≥2, and n is a positive integer;

[0043] The standard output image corresponding to the mixed light source image is generated based on the mixed light source image and the target chromaticity value.

[0044] Implementing the method provided in the second aspect, in some embodiments, the target chromaticity value is generated based on the chromaticity value corresponding to each calibration image in n calibration images and the fusion ratio corresponding to each calibration image, and there is a one-to-one correspondence between the n calibration images and the n light sources.

[0045] In some embodiments of the method provided in the second aspect, any one of the n calibration images is generated by preprocessing a single light source image corresponding to any one of the calibration images, and the preprocessing may include lens shading correction processing, image format conversion, and white balance calibration processing;

[0046] The shooting scene corresponding to the single light source image corresponding to any one of the calibration images has a light source corresponding to the any one of the calibration images.

[0047] Implementing the method provided by the second aspect, in some embodiments, the calibration image is a PNG image;

[0048] The fusion ratio corresponding to each calibration image is generated based on the G channel value of each calibration image.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the method described in the first aspect and any possible implementation method of the first aspect is executed.

[0050] In a fourth aspect, an embodiment of the present application provides a chip, comprising a logic circuit and an interface, wherein the logic circuit and the interface are coupled; the interface is used to input and / or output code instructions, and the logic circuit is used to execute the code instructions so that the method in the first aspect or any possible implementation of the first aspect is executed.

[0051] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a target terminal, execute the method described in the first aspect and any possible implementation of the first aspect.

[0052] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0053] It can be understood that the electronic devices provided in the second and third aspects, the chip provided in the fourth aspect, the computer-readable storage medium provided in the fifth aspect, and the computer program product provided in the sixth aspect are all related to the image processing method based on color adaptation provided in the first aspect, and can be used to execute the method provided in the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the image processing method based on color adaptation in the corresponding first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 It is a flow chart of an image color restoration method provided by the prior art;

[0056] Figure 2 It is a flowchart of an image processing method based on color adaptation provided in an embodiment of the present application;

[0057] Figure 3 is a schematic diagram of a scene for cutting a first processed image provided by an embodiment of the present application;

[0058] Figure 4 is a schematic diagram of a scene for generating a target image provided by an embodiment of the present application;

[0059] Figure 5a-5c A schematic diagram of an image processing scenario based on chromatic adaptation provided in an embodiment of the present application;

[0060] Figure 6 is a flowchart of another image processing method based on chromatic adaptation provided in an embodiment of the present application;

[0061] Figure 7 is a schematic diagram of a process for training a local white balance model provided in an embodiment of the present application;

[0062] Figure 8 It is a schematic diagram of a process of preparing training data for training a local white balance model provided by an embodiment of the present application;

[0063] Fig. 9 This is a schematic diagram of a scene for acquiring a single light source image provided by an embodiment of the present application;

[0064] Fig.10is another flowchart of preparing training data for training a local white balance model provided by an embodiment of the present application;

[0065] Fig.11 It is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0066] Fig.12 is a schematic diagram of the hardware structure of an electronic device 100 according to an embodiment of the present application;

[0067] Fig.13 A software structure block diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear contrary indication in the context. It should also be understood that in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a kind of association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B, can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0069] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0070] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0071] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given:

[0072] (1) RAW images

[0073] RAW image is the raw data that the image sensor converts the captured light source signal into a digital signal, and is the data format of the image output by the image sensor. A sensor can be called a photosensitive element, which is a device that converts an optical image into an electronic signal, such as a charge coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In the present application, the image sensor can be arranged in an image acquisition module. In the field of image technology, the RAW format is an unprocessed and uncompressed format, that is, the original image encoding data (digital negative). Common suffixes of the RAW format include .ARW, .SRF, .SR2, .crw, .cr2, .cr3, etc.

[0074] (2) PNG images

[0075] PNG format is a lossless compression image format, the full name is "Portable Network Graphics", that is, "Portable Network Graphics" or "Portable Network Graphics". As a standard format for transmitting images on the Internet, PNG is recommended by the World Wide Web Consortium (W3C) and is widely used on the Web. The PNG format supports three color schemes: index, grayscale, and RGB, as well as Alpha channels, which can achieve transparent effects on images.

[0076] (3) RGB color mode

[0077] The RGB color model is a color standard in the industry. It obtains a variety of colors by changing the three color channels of red (Red, R), green (Green, G), and blue (Blue, B) and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue. This standard covers almost all colors that can be perceived by human vision and is one of the most widely used color systems.

[0078] (4) Automatic white balance

[0079] Auto White Balance (AWB) refers to the use of an algorithm to restore the white imaged under ambient light of different color temperatures to true white (usually the white observed by people under natural daylight). Since the sensor cannot change its photosensitive characteristics according to changes in ambient light like the human eye, an additional module is needed to simulate human perception characteristics and restore the white image under different ambient light to true white. The AWB algorithm is mainly divided into two types: local white balance and global white balance.

[0080] (5) GT images

[0081] In machine learning, Ground Truth is generally used to represent the classification accuracy of the supervised learning training set, which is used to prove or disprove a hypothesis. Supervised machine learning will mark the training data, and the correctly marked data is the Ground Truth. In the embodiment of the present application, the GT image can be understood as the standard output image.

[0082] See also Figure 1 , Figure 1 A flowchart of an image color restoration method provided by the prior art.

[0083] like Figure 1 As shown, the method includes:

[0084] S101: Receive screen color gamut RGB data generated by a shooting device.

[0085] S102: Acquire the color temperature of the shooting environment when the shooting device shoots the image, the color temperature of the display environment of the display device, and the color coordinates of white in the screen color gamut information of the display device.

[0086] S103: Performing color adaptation transformation on the screen color gamut RGB data according to the shooting environment color temperature, the display environment color temperature and the color coordinates of white in the screen color gamut information of the display device to obtain the display environment RGB data.

[0087] Furthermore, the shooting device converts the sensor RGB data obtained when shooting the image into the color gamut of the display device according to the screen color gamut information, and obtains the screen color gamut RGB data.

[0088] Among them, performing color adaptation transformation on the screen color gamut RGB data according to the shooting environment color temperature, the display environment color temperature and the color coordinates of white in the screen color gamut information of the display device includes:

[0089] The shooting environment color temperature and the display environment color temperature are fused to obtain a fused color temperature;

[0090] Mapping the blended color temperature to the target color temperature;

[0091] Calculate the color coordinates corresponding to the target color temperature;

[0092] Calculate the color coordinates corresponding to the target color temperature and the color coordinate ratio matrix of white in the screen color gamut information;

[0093] Convert the screen color gamut RGB data into an XYZ color space to obtain first XYZ color data;

[0094] Convert the first XYZ color data into an LMS color space to obtain first LMS color data;

[0095] Multiplying the first LMS color data by the color coordinate ratio matrix to obtain second LMS color data;

[0096] Convert the second LMS color data into an XYZ color space to obtain second XYZ color data;

[0097] The second XYZ data is converted into the RGB color space to obtain display environment RGB data.

[0098] Furthermore, according to the screen color gamut information, converting the sensor RGB data obtained when the camera captures the image into the color gamut of the display device includes:

[0099] Calculate the color gamut mapping matrix according to the screen color gamut information;

[0100] The sensor RGB data is converted to the color gamut of the display device according to the color gamut mapping matrix.

[0101] It can be seen that in order to reproduce the viewing effect of the human eye when shooting images, the prior art selects reference shooting environment color temperature, display environment color temperature and screen color temperature (the color coordinates of white in the screen color gamut information represent the screen color temperature) to achieve the effect of accurately restoring the color of the image. The prior art can change the color of the image under different color temperature conditions by performing color adaptation transformation according to the shooting environment color temperature, the display environment color temperature and the color coordinates of white in the screen color gamut information, thereby improving the accuracy of image color restoration to achieve the visual effect when shooting in different viewing environments. However, the prior art does not take into account the distribution of light sources in the shooting scene, nor the complexity of the light received by each object in the image. It only performs color adaptation processing on the image based on the difference between the ambient color temperature value and the screen color temperature value, which easily leads to the problem of unclear object edges in the image after color adaptation processing, reduces the image quality, and fails to provide users with a good shooting experience.

[0102] Different from the prior art, the method of the embodiment of the present application fully considers the light conditions of each object in the image, and can perform color adaptation processing on each image block of the first processed image, which helps to ensure the color adaptation effect of each image block, thereby ensuring the imaging effect of each object in the image. In addition, the method of the embodiment of the present application can adopt the form of multiple processes to perform color adaptation processing on each image block at the same time, which helps to improve the image color adaptation processing effect while ensuring the rate of color adaptation processing, thereby further improving the user's shooting experience.

[0103] See also Figure 2 , Figure 2 A flowchart of an image processing method based on chromatic adaptation provided in an embodiment of the present application.

[0104] like Figure 2 As shown, the method may include:

[0105] S201: The electronic device 100 acquires a RAW image in response to a shooting instruction from a user.

[0106] It should be noted that the electronic device 100 may also be referred to as user equipment (UE for short). It may be handheld or wearable. It may also be referred to as a user terminal, terminal equipment, access terminal equipment, UE unit, UE station, mobile device, UE terminal equipment, mobile terminal, wireless communication equipment, UE agent or UE device, etc. Its specific form may be a mobile phone, a tablet computer (Pad), a computer with wireless transceiver function, a wearable terminal device, etc. It is an entity on the user side for receiving or transmitting signals. The specific structure of the electronic device 100 can be referred to. Fig.11 and Fig.12The relevant content will not be elaborated here.

[0107] S202: The electronic device 100 performs white balance processing on the RAW image to generate a first processed image corresponding to the RAW image.

[0108] In some possible implementations, the electronic device 100 may input the RAW image into a local white balance model, thereby outputting a first processed image.

[0109] Among them, the local white balance model in the embodiment method of the present application is a model obtained by training with mixed light source images as training samples and standard output images corresponding to the mixed light source images as labels. It helps to solve the problem of color imbalance in images captured in mixed light source scenes, and avoid serious color cast in the images finally presented to the user (such as some objects in the image appearing bluish or yellowish), thereby improving the user's shooting experience.

[0110] It should be noted that the format of the first processed image can be determined based on the image presentation requirements of the electronic device 100 (or the user's requirements), and the first processed image can be any one of JPEG, BMP, PNG or TITF, which is not limited in this application.

[0111] Furthermore, the image after white balance processing (such as the first processed image mentioned above) will carry the gain values ​​corresponding to the R channel, the G channel and the B channel. By way of example, there are four main arrangements of RGB data corresponding to the RAW image collected by the electronic device 100: GRBG, RGGB, BGGR and GBRG. It can be seen that the above four arrangements will result in one pixel corresponding to two G channel values. In order to facilitate the subsequent optimization of the image, the electronic device 100 can merge the two G channel values ​​corresponding to each pixel in the RAW image into one G value. The specific merging method can be to calculate the average value of the two G channel values. By way of example, if pixel A corresponds to G1 and G2 in the RAW image, after the electronic device 100 converts the RAW image into a PNG image, the G channel corresponding to pixel A in the PNG image can be (G1+G2) / 2.

[0112] S203: The electronic device 100 cuts the first processed image into i image blocks.

[0113] Wherein, i≥2, i is a positive integer. In order to retain the clarity of the edges of each object in the first processed image and achieve a better color adaptation processing effect, the method of the embodiment of the present application chooses to perform color adaptation processing at the full pixel level on the first processed image. Since performing color adaptation processing at the full pixel level on a single image will increase the amount of calculation of the electronic device 100, resulting in a long image processing time, in order to solve this problem, the method of the embodiment of the present application chooses to cut the first processed image into multiple image blocks, and simultaneously performs color adaptation processing on each image block, which helps to improve the efficiency of color adaptation processing.

[0114] Specifically, the electronic device 100 can cut the first processed image based on the i value pre-set by the technician, wherein the cutting method can be horizontal cutting or vertical cutting. Further, the electronic device 100 can determine the number of pixels of the width of the first processed image as the i value, and the electronic device 100 can also determine the number of pixels of the height of the first processed image as the i value, and then cut the first processed image based on the i value. Furthermore, the electronic device 100 can also cut the first processed image into i image blocks of size a×b, and the a and b values ​​can be determined based on the size of the first processed image and the i value. It should be noted that the units of a and b can be image size units such as inches, pixels and centimeters.

[0115] For example, see Figure 3 , Figure 3 A schematic diagram of a scene for cutting a first processed image is provided for an embodiment of the present application. Figure 3 As shown, the electronic device 100 in the method of the embodiment of the present application can cut the first processed image based on the size of the first processed image (the number of pixels corresponding to the image width and the number of pixels corresponding to the image height). Figure 3 The corresponding size of the first processed image in the image is 8 pixels × 6 pixels. The first processed image can be horizontally cut into 6 image blocks with a size of 8 pixels × 1 pixel according to the number of pixels corresponding to the image height, or the first processed image can be vertically cut into 8 image blocks with a size of 1 pixel × 6 pixels according to the number of pixels corresponding to the image width, thereby achieving the purpose of improving the color adaptation processing efficiency.

[0116] In some possible implementations, a technician may preset the value of i, that is, it can be considered that the electronic device 10 selects to cut the first processed images of various sizes into i image blocks. Exemplarily, let i be 24. If the electronic device 100 is capable of capturing images of multiple different sizes (e.g., it can capture images of two sizes, 1080 pixels × 1080 pixels and 3024 × 4032 pixels), when the image size corresponding to the user's shooting instruction is 1080 × 1080, the electronic device 100 can cut the first processed image (corresponding to a size of 1080 pixels × 1080 pixels) horizontally into 24 image blocks of 1080 pixels × 45 pixels, or vertically into 24 image blocks of 45 pixels × 1080 pixels; when the image size corresponding to the user's shooting instruction is 3024 × 4032, the electronic device 100 can cut the first processed image (corresponding to a size of 3024 × 4032) horizontally into 24 image blocks of 4032 × 126, or vertically into 24 image blocks of 168 pixels × 3024 pixels.

[0117] Optionally, when cutting the first processed image into multiple image blocks, the electronic device 100 may set a serial number label for each image block to indicate the position of each image block in the first processed image, so as to ensure that the second image block can be correctly spliced ​​subsequently.

[0118] It should be noted that the above examples of the method of cutting the first processed image and the specifications of the pixel matrix corresponding to the first processed image are only for illustrating the embodiment method of the present application in more detail, and should not constitute a limitation on the present application. Technicians can choose other cutting schemes based on actual conditions.

[0119] S204: The electronic device 100 performs color adaptation processing on each image block in parallel based on the color information of each image block in the i image blocks to generate a second processed image corresponding to each image block.

[0120] The electronic device 100 performs color adaptation processing on each of the i image blocks in parallel in a multi-process form to generate a second processed image corresponding to each image block.

[0121] Specifically, based on the color information of each image block, the color adaptation scheme corresponding to each image block is determined, and the color adaptation processing is performed on each image block in parallel. Exemplarily, the electronic device 100 can search for the corresponding target color value in the color adaptation table based on the color information corresponding to the image block, and determine the color direct gain matrix based on the image color value corresponding to each pixel in the image block, so as to achieve the purpose of performing color adaptation processing on the image block.

[0122] In some possible implementations, performing color adaptation processing on each image block in the i image blocks to generate a second processed image corresponding to each image block may include:

[0123] Based on the color information of the pixels contained in each image block, determining the chromatic adaptation gain matrix corresponding to each image block;

[0124] Based on the chromatic adaptation gain matrix corresponding to each pixel block, a second processed image corresponding to each image block is generated.

[0125] The color information may include a color temperature value, a brightness coefficient, and a chromaticity value.

[0126] Specifically, the electronic device 100 can directly query the chromaticity value of the color block corresponding to each image block, so as to determine at least one of the target color temperature value, target brightness coefficient or target chromaticity value corresponding to the image block. Further, based on at least one of the target color temperature value, target brightness coefficient or target chromaticity value, the chromatic adaptation gain value corresponding to each pixel point in the image block is determined, so as to determine the chromatic adaptation gain matrix corresponding to the image block. Furthermore, by multiplying the pixel point matrix corresponding to each image block with the chromatic adaptation gain matrix, a second processed image corresponding to each image block can be generated.

[0127] S205: The electronic device 100 stitches the second processed images corresponding to the image blocks to generate a target image.

[0128] For example, see Figure 4 , Figure 4 A schematic diagram of a scenario for generating a target image provided in an embodiment of the present application.

[0129] like Figure 4 As shown, each image block can generate a corresponding second processed image after color adaptation processing, and the electronic device 100 can splice the second processed image based on the serial number tags corresponding to each second image block (which can also be understood as the serial number tags corresponding to each image block). Optionally, the serial number tags can take the form of numbers, and the target image is generated by correctly splicing each second processed image according to the ascending rule. Among them, the target image can be understood as the image finally presented to the user.

[0130] See also Figure 5a-5c , Figure 5a-5c A schematic diagram of an image processing scenario based on color adaptation provided in an embodiment of the present application.

[0131] like Figure 5a As shown, after receiving the user's shooting instruction, the electronic device 100 obtains the corresponding RAW image, performs white balance and color adaptation processing on the RAW image, and finally generates a target image.

[0132] like Figure 5b As shown, the electronic device 100 can present the target image in the preview box of the shooting interface. Further, the electronic device 100 can present the target image in the relevant interface of the album application in response to the user's viewing instruction (such as clicking the preview box), which helps the user to view the target image more clearly and carefully.

[0133] Specifically, the processing process of the electronic device 100 on the RAW image can refer to Figure 5c .like Figure 5c As shown, a white balance operation is performed on the RAW image through a local white balance model, and a first processed image is output. Further, based on the size of the first processed image, the first processed image is horizontally cut into i image blocks, and then a multi-process synchronous execution method is used to perform color adaptation processing on each image block, so as to obtain a second processed image corresponding to each image block. Furthermore, the electronic device 100 can correctly stitch each second processed image into a target image based on the position tags corresponding to each second processed image.

[0134] It can be seen that the electronic device 100 in the embodiment of the present application can obtain a RAW image in response to the user's shooting instruction, and perform white balance processing on the RAW image to generate a first processed image, which helps to balance the color temperature and brightness of the image, etc., and prevent the image presented to the user from being partially overexposed or having severe local color cast (such as bluish or yellowish), which helps to ensure the user's shooting experience. Furthermore, the electronic device 100 can take corresponding color adaptation processing based on the corresponding light conditions of each part (or each pixel) of the first processed image, which helps to improve the effect of the image color adaptation processing, thereby providing the user with an image with better picture quality, so that the user has a better shooting experience.

[0135] For example, see Figure 6 , Figure 6 A flowchart of another image processing method based on color adaptation provided in an embodiment of the present application. Figure 6 As shown, the method of the embodiment of the present application first uses a mixed light source image (an image in RAW format, and there are n light sources in the shooting scene corresponding to the mixed light source image) and a standard output image corresponding to the mixed light source image (i.e. Figure 6 The mixed light source GT in the training local white balance model (i.e. Figure 6 The specific training process of the local white balance model can be referred to Figure 7-Figure 9 The relevant content will not be elaborated here.

[0136] Furthermore, after the local white balance model is trained, the local white balance model can be loaded into the electronic device 100. After the electronic device 100 obtains a RAW image in response to the user's shooting instruction, the electronic device 100 can input the RAW image into the local white balance model, and correspondingly, the local white balance model will output a LocalAWB inference result image (i.e., the first processed image mentioned above).

[0137] Furthermore, the electronic device 100 will perform color adaptation (CA) processing on the LocalAWB reasoning result map. Among them, the electronic device 100 will adopt the form of multiple processes to synchronously perform CA processing on each image block corresponding to the LocalAWB reasoning result map. Optionally, the electronic device 100 can refer to the CA processing method corresponding to the global white balance (Global AWB) to perform CA processing on each image block. Specifically, the CA processing method corresponding to the global white balance (Global AWB) can be understood as the electronic device 100 determining a color adaptation gain matrix (CA Gain Map) based on an image, and performing CA processing on the image based on the color adaptation gain matrix. Among them, the method of the embodiment of the present application helps to maintain the edge clarity of each object contained in the RAW image, ensure the imaging effect of each object in the image, and thus improve the color adaptation effect by determining the color adaptation gain matrix corresponding to each image block, and performing CA processing on each image block based on the color adaptation matrix corresponding to each image block.

[0138] Furthermore, the electronic device 100 performs color adaptation on each image block and obtains a second processed image (i.e., Figure 6 The target image can be stitched together with the result after the middle color adaptation to generate the target image, and finally the target image can be presented to the user in response to the user's viewing instruction.

[0139] See also Figure 7 , Figure 7 A flowchart of a method for training a local white balance model provided in an embodiment of the present application.

[0140] like Figure 7 Said method comprises:

[0141] S701: Acquire a training data set, where the training data set includes m mixed light source images and n single light source images corresponding to each of the m mixed light source images.

[0142] There are n light sources in the shooting scene corresponding to the mixed light source image, and there is a one-to-one correspondence between the single light source image and the light source, n≥2, n is a positive integer, and m≥2, m is a positive integer.

[0143] S702: Determine a target chromaticity value corresponding to each mixed light source image based on the chromaticity values ​​of n single light source images corresponding to each mixed light source image.

[0144] In some possible implementations, determining the target chromaticity value corresponding to each mixed light source image based on the chromaticity values ​​of n single light source images corresponding to each mixed light source image may include:

[0145] Preprocessing the n single light source images corresponding to each mixed light source image to generate n calibration images corresponding to each mixed light source image, where the calibration images have a one-to-one correspondence with the single light source images, and the preprocessing includes lens shading correction processing, image format conversion, and white balance calibration processing;

[0146] Based on the chromaticity value of each calibration image in the n calibration images corresponding to each mixed light source image and the fusion ratio corresponding to each calibration image, the target chromaticity value corresponding to each mixed light source image is determined. Optionally, the calibration image may be a PNG image.

[0147] Among them, the method of the embodiment of the present application can determine the fusion ratio of each calibration image corresponding to each mixed light source image based on the G channel value of each calibration image corresponding to each mixed light source image.

[0148] S703: Based on each mixed light source image and the target chromaticity value corresponding to each mixed light source image, generate a standard output image corresponding to each mixed light source image.

[0149] Among them, see Figure 8 , Figure 8 A schematic diagram of a process for preparing training data for training a local white balance model provided in an embodiment of the present application.

[0150] The method of the embodiment of the present application can use the mixed light source image and the standard output image corresponding to the mixed light source image as a training data pair (it can be understood that the local white balance model is a model trained with the mixed light source image as a training sample and the standard output image corresponding to the mixed light source image as a label), wherein the method of the embodiment of the present application can produce the standard output image corresponding to the mixed light source image based on each single light source image corresponding to any mixed light source image.

[0151] like Figure 8 As shown, the process of training the local white balance model may include:

[0152] First, for any mixed light source image, obtain each single light source image corresponding to the mixed light source image, where there are n light sources in the shooting scene corresponding to the mixed light source image, n≥2, and n is a positive integer. It should be noted that the above mixed light source image and the single light source images corresponding to the mixed light source image are all RAW images.

[0153] For example, see Fig. 9 , Fig. 9 A schematic diagram of a scene for acquiring a single light source image provided in an embodiment of the present application. Fig. 9 As shown, it is assumed that there is a shooting scene corresponding to the first mixed light source image (such as Fig. 9 There are three light sources (such as Fig. 9 The first light source 92, the second light source 93 and the third light source 94 in the embodiment of the present application can respectively obtain the corresponding single light source images for these three light sources (such as Fig. 9 It can be seen from the first single light source image 95, the second single light source image 96 and the third single light source image 97 in the figure that there is a one-to-one correspondence between the above multiple single light source images and the multiple shooting scenes. The method of the embodiment of the present application helps to subsequently generate a standard output image that is more in line with the color adjustment scheme of the mixed light source image by obtaining each single light source image corresponding to the mixed light source image, thereby helping to improve the training effect of the local white balance model.

[0154] Furthermore, the method of the embodiment of the present application can pre-process each single light source image corresponding to the mixed light source image to achieve the purpose of correcting the color imbalance problem existing in the single light source image. Optionally, the pre-processing can include lens shading correction processing, image format conversion processing and white balance calibration processing.

[0155] Specifically, in order to solve the problem of uneven image brightness distribution caused by the optical characteristics of the correction lens and the image sensor, the method of the embodiment of the present application can perform lens shading correction (Lens Shading Correction, LSC) on each single light source image and generate a corrected image corresponding to each single light source image, which helps to make the corrected image (which can be understood as the above-mentioned second processed image) have the same brightness level in the entire field of view.

[0156] Furthermore, the method of the embodiment of the present application can perform a format conversion operation on the corrected image, which helps to determine the RGB channel values ​​corresponding to each pixel point, and facilitates the subsequent calculation of the target brightness value. Optionally, the method of the embodiment of the present application can convert the corrected image in RAW format into a corresponding PNG image. The method of the embodiment of the present application converts the RAW image into a three-channel PNG image, so that the two G channel values ​​in the RAW image are merged into one G channel value, which helps to calculate the target chromaticity value later. The specific process of image format conversion can refer to the above Figure 2 The contents of step S202 and related embodiments are not described in detail here.

[0157] Furthermore, the method of the embodiment of the present application can perform white balance calibration processing on the PNG images corresponding to each corrected image, which helps to further balance the color of the image, reduce the color cast of the image, and make the color of each object in the image more consistent with the actual situation.

[0158] Furthermore, the method of the embodiment of the present application can generate a standard output image corresponding to the mixed light source image based on the mixed light source image and the target brightness value corresponding to the mixed light source image. The target chromaticity value can be generated based on the chromaticity value corresponding to each calibration image and the fusion ratio corresponding to each calibration image. Specifically, the fusion ratio corresponding to each calibration image is generated based on the G channel value of each calibration image.

[0159] For example, in combination Fig. 9 , pre-process the above three single light source images (i.e., the first single light source image 95, the second single light source image 96, and the third single light source image 97), and generate a first calibration image 951 (assuming that the corresponding G channel value is G 951 , the corresponding chromaticity value is L 951 ), the second calibration image 961 (assuming the corresponding G channel value is G 961 , the corresponding chromaticity value is L 961 ) and the third calibration image 971 (assuming the corresponding G channel value is G 971 , the corresponding chromaticity value is L 971 ).

[0160] Furthermore, it can be determined that the fusion weight corresponding to the first calibration image 951 is W 951 =G 951 / (G 951 +G 961 +G971, the fusion weight corresponding to the second calibration image 961 is W961=G961 / G951+G961+G971, and the fusion weight corresponding to the third calibration image 971 is W 971 =G 971 / (G 951 +G 961 +G 971 ), and then the target chromaticity value can be determined to be L = L 951 ×W 951 +L 961 ×W 961 +L 971 ×W 971 Furthermore, the method of the embodiment of the present application can generate a standard output image corresponding to the mixed light source image based on the mixed light source image and the target chromaticity value corresponding to the mixed light source image.

[0161] It should be noted that in the process of training the local white balance model according to the method of the embodiment of the present application, there are several mixed light source images. The embodiment of the present application is only to explain in detail the main training steps of training the local white balance model, so only a single mixed light source image is used as an example for illustration. Therefore, the above example should not constitute a limitation to the present application.

[0162] It can be seen that the local white balance model in the implementation method of the present application uses mixed light source images as training samples, which helps the local white balance model to cover more image shooting scenes, ensures that images collected or shot in various complex light source scenes can have good color balancing capabilities, and lays the foundation for subsequent color adaptation processing, thereby ensuring the user's shooting experience. Furthermore, the embodiment method of the present application can generate training labels for the local white balance model based on the mixed light source and the target chromaticity value, and through self-made training data pairs, it helps to ensure the iterative effect of the local white balance model and improve the color balancing ability of the local white balance model.

[0163] See also Fig.10 , Fig.10 A schematic diagram of another process for preparing training data for training a local white balance model provided in an embodiment of the present application.

[0164] like Fig.10 As shown, based on a single mixed light source image, the method of the embodiment of the present application can perform lens shading correction processing, conversion to three-channel PNG (i.e. the above-mentioned format conversion operation) and white balance calibration processing on each single light source image corresponding to the mixed light source image. The method of the embodiment of the present application can also perform lens shading correction processing, conversion to three-channel PNG (i.e. the above-mentioned format conversion operation) and white balance calibration processing on each single light source image corresponding to the mixed light source image. Fig.10 G1, G2 and G n ), determine the fusion ratio corresponding to each calibration image (such as Fig.10 W1, W2 and W n ).

[0165] Specifically, the calculation process of the fusion ratio can refer to the following calculation process:

[0166] W1=G1 / (G1+G2+…+G n )W2=G2 / (G1+G2+…+G n )....W n =G n / (G1+G2+…+G n )

[0167] Furthermore, the method of the embodiment of the present application can be based on the chromaticity value corresponding to each image block (such as Fig.10 L1, L2 and L n ) and the fusion ratio to generate the target chromaticity value corresponding to the mixed light source image.

[0168] Specifically, the calculation process of the target chromaticity value can refer to the following calculation process:

[0169] L 1～n =L1×W1+L2×W2+…+L n ×W n

[0170] Furthermore, the method of the embodiment of the present application can generate a standard output image corresponding to the mixed light source (ie, Fig.10 Mixed light source GT).

[0171] S704: Using each mixed light source image as training data and using the standard output image corresponding to each mixed light source image as a label, a neural network is trained to obtain a local white balance model.

[0172] It should be noted that Figure 7 The trained local white balance model can be used to implement Figure 2-Figure 6 The corresponding image processing method based on color adaptation and its related embodiment contents.

[0173] See also Fig.11 , Fig.11 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application.

[0174] like Fig.11 As shown, the electronic device may include: an interaction module 810, an image acquisition module 820 and an image processing module 830;

[0175] The interaction module 810 may be used to receive a user's shooting instruction;

[0176] The image acquisition module 820 may be used to acquire a RAW image in response to a user's shooting instruction;

[0177] The image processing module 830 may be used to perform white balance processing on the RAW image to generate a first processed image corresponding to the RAW image;

[0178] The image processing module 830 may also be used to cut the first processed image into i image blocks, where i≥2 and i is a positive integer;

[0179] The image processing module 830 may also be used to perform color adaptation processing on each of the i image blocks in parallel based on the color information of each of the i image blocks to generate a second processed image corresponding to each of the image blocks;

[0180] The image processing module 830 may also be used to stitch the second processed images corresponding to the image blocks together to generate a target image.

[0181] In some possible implementations, the electronic device may further include:

[0182] The image processing module 830 may also be used to determine the chromatic adaptation gain matrix corresponding to each image block based on the color information of the pixels contained in each image block, where the color information may include a color temperature value, a brightness coefficient, and a chromaticity value;

[0183] The image processing module 830 may also be used to generate a second processed image corresponding to each image block based on the chromatic adaptation gain matrix corresponding to each pixel block.

[0184] In some other possible implementations, the electronic device may further include:

[0185] The image processing module 830 may also be used to perform color adaptation processing on each image block in parallel in the form of multiple processes.

[0186] In some other possible implementations, the electronic device may further include:

[0187] The image processing module 830 may also be used to input the RAW image into the local white balance model and output a first processed image.

[0188] In some other possible implementations, the local white balance model is a model obtained by training using a mixed light source image as a training sample and a standard output image corresponding to the mixed light source image as a label;

[0189] There are n light sources in the shooting scene corresponding to the mixed light source image, n≥2, and n is a positive integer;

[0190] The standard output image corresponding to the mixed light source image is generated based on the mixed light source image and the target chromaticity value.

[0191] In some other possible implementations, the target chromaticity value is generated by fusing the chromaticity value corresponding to each calibration image in the n calibration images and the fusion ratio corresponding to each calibration image, and there is a one-to-one correspondence between the n calibration images and the n light sources.

[0192] In some other possible implementations, any one of the n calibration images is generated by preprocessing a single light source image corresponding to any one of the calibration images, and the preprocessing may include lens shading correction processing, image format conversion, and white balance calibration processing;

[0193] The shooting scene corresponding to the single light source image corresponding to any one of the calibration images has a light source corresponding to the any one of the calibration images.

[0194] In some other possible implementations, the calibration image is a PNG image;

[0195] The fusion ratio corresponding to each calibration image is generated based on the G channel value of each calibration image.

[0196] For further information, see Fig.12 , Fig.12 A schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application is shown.

[0197] The electronic device 100 may include a processor 101, a memory 102, a wireless communication module 103, a mobile communication module 104, an antenna 103A, an antenna 104A, a power switch 105, a sensor module 106, a focus motor 107, a camera 108, a display screen 109, etc. Among them, the sensor module 106 may include a gyroscope sensor 106A, an acceleration sensor 106B, an ambient light sensor 106C, an image sensor 106D, a distance sensor 106E, etc. Among them, the wireless communication module 103 may include a WLAN communication module, a Bluetooth communication module, etc. The above multiple parts can transmit data through a bus.

[0198] The processor 101 may include one or more processing units, for example, the processor 101 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0199] The memory 102 may be used to store computer executable program codes, and the executable program codes may include instructions. The processor 101 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 102. The memory 102 may include a program storage area and a data storage area. In a specific implementation, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0200] The wireless communication function of the electronic device 100 can be implemented through the antenna 103A, the antenna 104A, the mobile communication module 104, the wireless communication module 103, the modem processor and the baseband processor.

[0201] Antenna 103A and antenna 104A can be used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas.

[0202] The mobile communication module 104 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 100 .

[0203] The wireless communication module 103 can provide wireless communication solutions including wireless local area networks (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. for application in the electronic device 100.

[0204] The gyro sensor 106A may be used to determine the motion posture of the electronic device 100 .

[0205] The acceleration sensor 106B can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes).

[0206] The electronic device 100 can realize the shooting function through the ISP, the camera 108, the video codec, the GPU, the display screen 109 and the application processor.

[0207] The electronic device 100 can realize the display function through a GPU, a display screen 109, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 109 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 101 may include one or more GPUs, which execute program instructions to generate or change display information.

[0208] The display screen 109 is used to display images, videos, etc. The display screen 109 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 109, where N is a positive integer greater than 1.

[0209] The structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0210] In the embodiments of the present application:

[0211] Any one or more of the wireless communication module 103 , the mobile communication module 104 , the sensor module 106 , the focus motor 107 , the camera 108 , etc. may be used to detect the current environmental characteristics of the electronic device 100 .

[0212] The display screen 109 is used to display the user interface provided by the electronic device 100, such as a shooting interface or an album interface. The user interface displayed by the display screen 109 can refer to the UI embodiment described above.

[0213] For details of the operations performed by the various components in the electronic device 100, please refer to the relevant description of the above method embodiments.

[0214] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes a mobile operating system with a layered architecture as an example to exemplify the software structure of the electronic device 100.

[0215] It is understandable that the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, and this application does not limit this. Fig.13 , Fig.13 This is a software structure diagram provided by the embodiment of the present application. Fig.13 The layered architecture shown divides the system into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom, namely, application layer, application framework layer, hardware abstraction layer, driver layer, and hardware layer.

[0216] The application layer may include a series of application packages. In the embodiment of the present application, the application package may include a camera, a gallery, etc.

[0217] The application framework layer provides an application programming interface (API) and a programming framework for the application of the application layer. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework layer may include a camera access interface, wherein the camera access interface may include camera management and camera devices. The camera access interface is used to provide an application programming interface and a programming framework for camera applications.

[0218] The hardware abstraction layer is an interface layer between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0219] The camera hardware abstraction layer may provide virtual hardware for camera device 1, camera device 2 or more camera devices. The camera algorithm library may include operating codes and data for implementing the shooting method provided in the embodiment of the present application.

[0220] The driver layer is a layer between hardware and software. The driver layer includes drivers for various hardware. The driver layer may include camera device drivers, digital signal processor drivers, and image processor drivers.

[0221] The camera device driver is used to drive the camera sensor to collect images and drive the image signal processor to pre-process the images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0222] The hardware layer is a layer located below the driver layer. The hardware layer includes sensors, image signal processors, digital signal processors, and image processors. Among them, the sensors may include different types of sensors 1, sensor 2, time of flight (TOF) sensors, and multi-spectral sensors. This application does not limit the type of sensors.

[0223] Specifically, in response to the user's operation of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera by calling the camera device (camera device 1 and / or other camera devices) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the kernel layer. The camera device driver can start the corresponding camera sensor and collect image light signals through the sensor. A camera device in the camera hardware abstraction layer corresponds to a camera sensor in the hardware layer.

[0224] Then, the camera sensor can transmit the collected image light signal to the image signal processor for preprocessing to obtain the image electrical signal (raw image), and transmit the above raw image to the camera hardware abstraction layer through the camera device driver.

[0225] The camera hardware abstraction layer may include a camera algorithm library that sends the original image. The camera algorithm library stores program code for implementing the shooting method provided in the embodiment of the present application. Based on a digital signal processor and an image processor, the camera algorithm library executes the above code to achieve the above-mentioned capabilities of object recognition and marking, protagonist tracking, and extraction of close-up images.

[0226] The camera algorithm library can send the original image captured by the camera to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can send it for display. At the same time, the camera algorithm library can also output the center point of the object in the recognized image frame and the close-up image centered on the protagonist. In this way, the camera application can display a selection box mark on the original image sent for display based on the center point of the object, and display the close-up image in a small window.

[0227] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0228] The present application also provides an electronic device, which may include: a memory and a processor. The memory may be used to store a computer program; the processor may be used to call the computer program in the memory so that the electronic device executes the method executed by the electronic device side in any of the above embodiments.

[0229] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the electronic device side in any of the above embodiments.

[0230] In some possible designs, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0231] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0232] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0233] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0234] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0235] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the electronic device side in any of the above embodiments.

[0236] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device side in any of the above embodiments.

[0237] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0238] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0239] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0240] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0241] In short, the above is only an embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of this application shall be included in the protection scope of this application.

Claims

1. An image processing method based on chromatic adaptation, characterized in that: The method comprises: Responding to a shooting instruction from a user, acquiring a RAW image; Performing white balance processing on the RAW image to generate a first processed image corresponding to the RAW image; Cutting the first processed image into i image blocks, i≥2, i is a positive integer; Based on the color information of each image block in the i image blocks, perform color adaptation processing on each image block in parallel to generate a second processed image corresponding to each image block; The second processed images corresponding to the image blocks are spliced ​​to generate a target image.

2. The method according to claim 1, characterized in that: The method of performing color adaptation processing on each image block in parallel based on color information corresponding to each image block in the i image blocks to generate a second processed image corresponding to each image block includes: Determine a chromatic adaptation gain matrix corresponding to each image block based on color information of pixels included in each image block, wherein the color information includes a color temperature value, a brightness coefficient, and a chromaticity value; Based on the chromatic adaptation gain matrix corresponding to each pixel block, a second processed image corresponding to each image block is generated.

3. The method according to claim 1 or 2, characterized in that: The performing color adaptation processing on each image block separately includes: In the form of multiple processes, color adaptation processing is performed on each image block in parallel.

4. The method according to claim 3, characterized in that The performing white balance processing on the RAW image to generate a first processed image corresponding to the RAW image includes: The RAW image is input into a local white balance model, and the first processed image is output.

5. The method according to claim 4, characterized in that The local white balance model is a model obtained by training with a mixed light source image as a training sample and a standard output image corresponding to the mixed light source image as a label; There are n light sources in the shooting scene corresponding to the mixed light source image, n≥2, and n is a positive integer; The standard output image corresponding to the mixed light source image is generated based on the mixed light source image and the target chromaticity value.

6. The method according to claim 5, characterized in that The target chromaticity value is generated based on the chromaticity value corresponding to each calibration image in the n calibration images and the fusion ratio corresponding to each calibration image, and there is a one-to-one correspondence between the n calibration images and the n light sources.

7. The method according to claim 6, characterized in that Any one of the n calibration images is generated by preprocessing a single light source image corresponding to the any one of the calibration images, wherein the preprocessing includes lens shading correction processing, image format conversion, and white balance calibration processing; The shooting scene corresponding to the single light source image corresponding to any one of the calibration images has a light source corresponding to the any one of the calibration images.

8. The method according to claim 7, characterized in that The calibration image is a PNG image; The fusion ratio corresponding to each calibration image is generated based on the G channel value of each calibration image.

9. A method for training a local white balance model, characterized in that: The method comprises: Acquire a training data set, the training data set comprising m mixed light source images and n single light source images corresponding to each of the m mixed light source images, there are n light sources in the shooting scene corresponding to the mixed light source image, and there is a one-to-one correspondence between the single light source images and the light sources, n≥2, n is a positive integer, and m≥2, m is a positive integer; Determine the target chromaticity value corresponding to each mixed light source image based on the chromaticity values ​​of n single light source images corresponding to each mixed light source image; Based on each mixed light source image and the target chromaticity value corresponding to each mixed light source image, a standard output image corresponding to each mixed light source image is generated; The mixed light source images are used as training data, and the standard output images corresponding to the mixed light source images are used as labels to train the neural network and obtain a local white balance model.

10. The method according to claim 9, characterized in that The step of determining the target chromaticity value corresponding to each mixed light source image based on the chromaticity values ​​of the n single light source images corresponding to each mixed light source image comprises: Preprocessing n single light source images corresponding to each mixed light source image to generate n calibration images corresponding to each mixed light source image, wherein the calibration images have a one-to-one correspondence with the single light source images, and the preprocessing includes lens shading correction processing, image format conversion, and white balance calibration processing; Based on the chromaticity value of each calibration image in the n calibration images corresponding to each mixed light source image and the fusion ratio corresponding to each calibration image, the target chromaticity value corresponding to each mixed light source image is determined.

11. The method according to claim 10, characterized in that The calibration image is a PNG image; The method further comprises: Based on the G channel values ​​of the calibration images corresponding to the mixed light source images, the fusion ratios of the calibration images corresponding to the mixed light source images are determined.

12. An electronic device, characterized in that: The electronic device comprises: an interaction module, an image acquisition module and an image processing module; The interaction module is used to receive a shooting instruction from a user; The image acquisition module is used to acquire a RAW image in response to a shooting instruction of the user; The image processing module is used to perform white balance processing on the RAW image to generate a first processed image corresponding to the RAW image; The image processing module is further used to cut the first processed image into i image blocks, i≥2, i is a positive integer; The image processing module is further used to perform color adaptation processing on each of the i image blocks in a multi-process form to generate a second processed image corresponding to each image block; The image processing module is further used to splice the second processed images corresponding to the respective image blocks to generate a target image.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions so that the method according to any one of claims 1 to 8 is executed.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images