Model training method, image processing method and electronic device

Abstract

The embodiment of the present application provides a model training method, an image processing method and an electronic device, which relate to the field of terminal technology. The method includes: inputting a sample image of at least one target color channel into an image processing model to obtain a first processed image output by the image processing model, and determining the hue parameters of the first processed image. According to the sample offset coefficient, the hue parameters of the first processed image are offset to obtain reference hue parameters. The sample image and the sample offset coefficient of at least one target color channel are input into the image processing model to obtain a second processed image output by the image processing model, and the actual hue parameters of the second processed image are determined. The model parameters of the image processing model are updated according to the reference hue parameters and the actual hue parameters. Thereafter, the hue offset coefficient and the original image are input into the trained image processing model to obtain the target image. In this way, the problem of image color cast can be solved simply and effectively.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a model training method, an image processing method and an electronic device. Background Art

[0002] Applications for implementing image capture functions are usually installed in terminal devices. These applications need to process the original images captured by the camera in order to obtain output images that can be displayed.

[0003] For example, you can deploy an image processing model in an application, and then use the image processing model to process the original image to obtain an output image. Ideally, the image processing performed by the image processing model on the original image will not change the hue information of the original image. Because the hue information of the original image is generally accurate, the hue of the final output image can be guaranteed to be normal.

[0004] However, during the model deployment process, some model accuracy may be lost for performance reasons, which may cause color cast problems in the final output image. Summary of the invention

[0005] The embodiment of the present application provides a model training method, an image processing method and an electronic device, which are applied to the field of terminal technology. By adding a hue shift coefficient of at least one target color channel to the input data of the image processing model, the image processing model can perform a corresponding degree of hue shift according to the hue shift coefficient during the image processing process, thereby specifically correcting the problem of color cast in the model-processed image.

[0006] In a first aspect, an embodiment of the present application proposes a model training method. The method comprises:

[0007] Determine a sample shift coefficient of at least one target color channel, where the sample shift coefficient is used to indicate that a hue parameter corresponding to the target color channel is shifted;

[0008] Inputting the sample image into an image processing model to obtain a first processed image output by the image processing model, and determining a tone parameter of the first processed image;

[0009] performing an offset process on the tone parameter of the first processed image according to the sample offset coefficient to obtain a reference tone parameter;

[0010] Inputting the sample image and the sample shift coefficient of the at least one target color channel into an image processing model to obtain a second processed image output by the image processing model, and determining an actual hue parameter of the second processed image;

[0011] The model parameters of the image processing model are updated according to the reference tone parameter and the actual tone parameter.

[0012] In this way, the image processing model can learn the following capabilities: in the image processing process, the hue parameters of the corresponding color channels are offset according to the input sample offset parameters. When the image processing model has such capabilities, the hue offset coefficient can be set according to the actual color cast situation during the model application stage, so that the image processing model can perform hue offset on the image according to the hue offset coefficient, thereby achieving color cast correction.

[0013] In a possible implementation manner, the color tone parameter is an automatic white balance (AWB) gain, and the AWB gain includes a gain parameter of at least one color channel;

[0014] The determining of the sample offset coefficient of at least one target color channel comprises:

[0015] For any of the target color channels, among the gain parameters of at least one color channel of the sample image, determine the gain parameter of the target color channel as a first initial gain parameter;

[0016] Performing numerical adjustment processing on the first initial gain parameter to obtain a first adjusted gain parameter of the target color channel;

[0017] A ratio of the first initial gain parameter to the first adjusted gain parameter is determined as a sample offset coefficient of the target color channel.

[0018] In a possible implementation, the performing numerical adjustment processing on the first initial gain parameter to obtain a first adjusted gain parameter of the target color channel includes:

[0019] Generate a first value within a preset value range, and perform the value adjustment process on the first initial gain parameter according to the first value to obtain a first adjusted gain parameter of the target color channel;

[0020] The numerical value adjustment process is randomly selected from the adjustment types, or the numerical value adjustment process is predetermined from the adjustment types, and the adjustment types include: increasing the numerical value, decreasing the numerical value, and keeping the numerical value unchanged.

[0021] In this implementation, the hue parameter can be numerically adjusted based on the sample image, so as to construct a virtual color cast scene to determine the sample shift coefficient. This implementation can ensure the rationality and practicality of the determined sample shift coefficient.

[0022] In a possible implementation, performing an offset process on the tone parameter of the first processed image according to the sample offset coefficient to obtain a reference tone parameter includes:

[0023] For any of the target color channels, among the gain parameters of at least one color channel of the first processed image, determining the gain parameter of the target color channel as a second initial gain parameter;

[0024] Determining a second adjustment gain parameter of the target color channel according to a product of the sample offset coefficient of the target color channel and the second initial gain parameter;

[0025] Among the gain parameters of at least one color channel of the first processed image, the gain parameter of the target color channel is replaced with the second adjustment gain parameter to obtain the reference hue parameter.

[0026] In this implementation, the labeled data, that is, the relevant data of the expected model output, can be effectively constructed, so that the model can be correctly fed back to effectively optimize the model.

[0027] In a second aspect, an embodiment of the present application provides an image processing method. The method includes:

[0028] Acquire an original image captured by a camera, and acquire a hue shift coefficient of at least one target color channel, wherein the hue shift coefficient is used to indicate that a hue parameter corresponding to the target color channel is to be shifted;

[0029] The original image and the hue shift coefficient of the at least one target color channel are input into an image processing model to obtain a target image output by the image processing model, wherein the image processing model is trained according to the method described in the first aspect.

[0030] In this way, by adding a hue shift coefficient to the input data to control the color tendency of the image output, the image processing model is specifically controlled to refer to the hue shift coefficient, and the hue parameters of the image are shifted to a corresponding degree during the image processing process, thereby compensating for the color cast of the image processing model to ensure that the target image finally output by the image processing model can correctly represent the hue in the actual environment.

[0031] In a possible implementation manner, obtaining a hue shift coefficient of at least one target color channel includes:

[0032] Inputting a test input image into an image processing model to obtain a test output image output by the image processing model;

[0033] The hue shift coefficient of the at least one target color channel is determined according to the hue parameter of the test input image and the hue parameter of the test output image. In one implementation, for example, for any target color channel, the ratio of the hue parameter of the target color channel in the test input image to the hue parameter of the target color channel in the test output image can be determined as the hue shift coefficient of the target color channel.

[0034] In this implementation, the model is tested to determine the hue shift coefficient of the target color channel according to the input and output of the model, thereby ensuring the correctness of the hue shift coefficient.

[0035] In a possible implementation, the processing unit in the image processing model includes at least one of the following:

[0036] A black level correction unit, an automatic white balance unit, an integrated processing unit, a local color correction unit and a lens shading correction unit, wherein the integrated processing unit is used to implement at least one of the following functions: noise reduction processing, demosaicing processing and high dynamic range processing.

[0037] The local color correction unit is used to perform color cast compensation on the intermediate image output by the integrated processing unit according to the compensation matrix.

[0038] In a possible implementation, the method further includes:

[0039] During the processing of the image processing model, obtaining an intermediate image output by the integrated processing unit;

[0040] Dividing the original image and the intermediate image into n image regions, wherein the n image regions in the original image correspond to the n image regions in the intermediate image in a one-to-one manner, and n is an integer greater than or equal to 1;

[0041] For any of the image regions, determining a regional compensation value of at least one target color channel corresponding to the image region according to first pixel value information of the image region in the original image and second pixel value information of the image region in the intermediate image;

[0042] A first compensation matrix of the at least one target color channel is determined according to the area compensation value of the at least one target color channel corresponding to each of the image areas.

[0043] In a possible implementation, determining a first compensation matrix of at least one target color channel according to the area compensation value of at least one target color channel corresponding to each of the image areas includes:

[0044] For any of the target color channels, the area compensation value of each of the image areas is set to the pixel compensation value corresponding to each pixel point in each of the image areas;

[0045] The first compensation matrix is ​​determined by a matrix composed of pixel compensation values ​​corresponding to each pixel point in the intermediate image.

[0046] In this implementation, by determining the first compensation matrix and then compensating the intermediate image according to the first compensation matrix, it is possible to perform local color cast compensation for each image area in the intermediate image, so as to further improve the hue correctness of the target image that is finally output.

[0047] In a possible implementation, after determining the first compensation matrix by forming a matrix composed of pixel compensation values ​​corresponding to each pixel point in each image area, the method further includes:

[0048] For a transition region between any two adjacent image regions, smoothing the pixel compensation values ​​of the pixel points in the transition region to obtain smoothed pixel compensation values ​​of the pixel points in the transition region;

[0049] The first compensation matrix is ​​updated according to the smoothed pixel compensation values ​​of the pixels in the transition area.

[0050] In this implementation, the pixel compensation values ​​of adjacent areas in the first compensation matrix are smoothed to ensure that after the local color cast compensation is performed, there will be no color tone jump problem.

[0051] In a possible implementation manner, the pixel value information includes an average pixel value of a first color channel of each pixel point in the image area, an average pixel value of a second color channel of each pixel point in the image area, and an average pixel value of a third color channel of each pixel point in the image area;

[0052] The determining, according to the first pixel value information of the image area in the original image and the second pixel value information of the image area in the intermediate image, a regional compensation value of at least one target color channel corresponding to the image area comprises:

[0053] Determine a ratio of an average pixel value of the first color channel to an average pixel value of the second color channel in the first pixel value information as a first ratio;

[0054] Determine a ratio of an average pixel value of the first color channel to an average pixel value of the second color channel in the second pixel value information as a second ratio;

[0055] The ratio of the first ratio to the second ratio is determined as the regional compensation value of the target color channel.

[0056] In this implementation, the regional compensation value of the target color channel is determined by determining the change in R:G (or B:G) of the image area in the original image and the R:G (or B:G) of the image area in the intermediate image, so as to ensure that the determined regional compensation value can achieve adaptive correction of the color cast.

[0057] In a possible implementation, the method further includes:

[0058] Acquire a preset second compensation matrix, wherein the matrix size of the second compensation matrix is ​​consistent with the image size of the intermediate image, and the compensation value of each matrix position in the second compensation matrix is ​​proportional to a first distance corresponding to the matrix position, and the first distance is the distance between the matrix position and the center position of the second compensation matrix;

[0059] Multiplying the first compensation matrix and the second compensation matrix at corresponding positions to obtain a target compensation matrix;

[0060] According to the target compensation matrix, compensation processing is performed on the pixel value of each pixel point of the intermediate image.

[0061] In this implementation, the effect of the second compensation matrix is ​​superimposed on the first compensation matrix to increase the compensation ratio of the four corners of the image, thereby effectively improving the image quality after color cast processing.

[0062] In a third aspect, an embodiment of the present application provides a model training device, which may be an electronic device, or a chip or chip system in an electronic device. The model training device may include a display unit and a processing unit. When the model training device is an electronic device, the display unit may be a display screen. The display unit is used to perform the display step so that the electronic device implements a model training method described in the first aspect or any possible implementation of the first aspect. When the model training device is an electronic device, the processing unit may be a processor. The model training device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements a model training method described in the first aspect or any possible implementation of the first aspect. When the model training device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements a model training method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (eg, a register, a cache, etc.), or a storage unit within the electronic device that is located outside the chip (eg, a read-only memory, a random access memory, etc.).

[0063] In a fourth aspect, an embodiment of the present application provides an image processing device, which may be an electronic device, or a chip or chip system in an electronic device. The image processing device may include a display unit and a processing unit. When the image processing device is an electronic device, the display unit may be a display screen. The display unit is used to perform the display step so that the electronic device implements an image processing method described in the second aspect or any possible implementation of the second aspect. When the image processing device is an electronic device, the processing unit may be a processor. The image processing device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the second aspect or any possible implementation of the second aspect. When the image processing device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the second aspect or any possible implementation of the second aspect. The storage unit may be a storage unit within the chip (eg, a register, a cache, etc.), or a storage unit within the electronic device that is located outside the chip (eg, a read-only memory, a random access memory, etc.).

[0064] In a fifth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the methods described in the first aspect and the second aspect.

[0065] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed on a computer, the computer executes the method described in the first aspect and the second aspect.

[0066] In a seventh aspect, an embodiment of the present application provides a computer program product comprising a computer program. When the computer program runs on a computer, the computer executes the methods described in the first and second aspects.

[0067] In an eighth aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect and the second aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0068] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0069] It should be understood that the second to eighth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of an image processing model provided in an embodiment of the present application;

[0071] Figure 2 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0072] Figure 3 A schematic diagram of the software structure of a terminal device provided in an embodiment of the present application;

[0073] Figure 4 A schematic diagram of the training process of the image processing model provided in the embodiment of the present application;

[0074] Figure 5A schematic diagram of the process of an image processing method provided in an embodiment of the present application;

[0075] Figure 6 A schematic diagram of an implementation of determining a hue shift coefficient provided in an embodiment of the present application;

[0076] Figure 7 Schematic diagram of the image processing model provided in the embodiment of the present application Figure 2 ;

[0077] Figure 8 A schematic diagram of the processing process of local color cast correction provided in an embodiment of the present application;

[0078] Fig. 9 A schematic diagram of determining a second compensation matrix provided in an embodiment of the present application;

[0079] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0081] 1. RAW images

[0082] RAW images (original images) refer to uncompressed and unprocessed image data. They are image data directly obtained by the camera without any image processing or color encoding. RAW images contain original pixel information, which means that no data is lost. Compared with other compressed formats such as JPEG, RAW images can retain a wider dynamic range, which is very useful when dealing with high-contrast scenes and post-image editing.

[0083] The advantage of this format is that it can save most of the captured image information, so that in post-processing, the user can reprocess the file at any time to get the desired effect, such as adjusting white balance, exposure compensation, tonal range, contrast, color saturation and sharpness. Moreover, when adjusting the camera raw image, the original camera raw data will be saved, so that the original image information can be retained to the greatest extent.

[0084] In general, RAW images are an important image format that can provide a wider dynamic range and richer color information, thus providing greater flexibility for post-processing.

[0085] 2. Gray World Method

[0086] The gray world method is a method for calculating AWB gain, wherein the gray world algorithm is based on a hypothesis: for any image, when it has sufficient color changes, the mean values ​​of its RGB components tend to be equal (ie, gray).

[0087] 3. AWB Gain

[0088] For a fixed image, its AWB gain (gain) can be fixedly calculated. The AWB gain may include an R channel gain (R_gain), a G channel gain (G_gain) and a B channel gain (B_gain).

[0089] Based on the grayscale world method introduced above, when calculating the AWB gain, the average pixel value of each color channel in the image can be calculated first, so as to obtain Rmean, Gmean and Bmean. And usually it is necessary to fix a certain color channel as a reference value, because the human eye is more sensitive to green, so the G color channel can be fixed, that is, Gmean is selected as the reference value, and the corresponding G_gain is set to 1.

[0090] And then, the channel gains are calculated for the remaining color channels, for example, R_gain=Rmean / Gmean and Bgain=Bmean / Gmean can be calculated.

[0091] 4. RBG

[0092] The RGB color mode is a color standard that obtains a variety of colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue.

[0093] 5. Other terms

[0094] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0095] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0096] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0097] 6. Electronic equipment

[0098] The electronic device of the embodiment of the present application may include a handheld device, a vehicle-mounted device, etc. with an image capture function. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 5G networks or future evolved public land mobile communication networks (PLANs). The embodiments of the present application do not limit this.

[0099] As an example but not limitation, in the embodiment of the present application, the electronic device may also be a wearable device. In addition, in the embodiment of the present application, the electronic device may also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0100] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0101] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0102] Based on the above introduction, the relevant technologies involved in this application are further introduced in detail below.

[0103] With the continuous development of terminal device technology, using terminal devices to capture images has become a common application scenario. For example, an application for realizing the image capture function can be installed in the terminal device, where the application can be a camera application built into the terminal device, or a third-party camera application, which is not limited in this embodiment.

[0104] Regardless of the application, as long as it can realize the image capture function, it can usually respond to the user's triggered capture operation and call the terminal device's camera to capture the image. After that, the application will process the original image captured by the terminal device's camera to obtain an output image that can restore the details of the scene.

[0105] In order to achieve the purpose of image processing, for example, an image processing model can be deployed in the application. After obtaining the original image captured by the camera, the image processing model can be used to process the original image to obtain an output image displayed in the terminal device.

[0106] Combine the following Figure 1 A brief introduction to possible implementations of image processing models is given. Figure 1 A schematic diagram of an image processing model provided in an embodiment of the present application.

[0107] Reference Figure 1 , the original image can be input into the image processing model. The image processing model may include the following units: a black level correction (BLC) unit, an automatic white balance (AWB) unit, an integrated processing unit, and a lens shading correction (LSC) unit.

[0108] Below Figure 1 Each processing unit shown is described in turn:

[0109] 1. Black level correction unit:

[0110] Due to the differences between the individual pixel elements in the image sensor and factors such as circuit noise, the reference value of the black level may be offset to a certain extent. If the black level is not corrected, the black part of the image may appear gray or have color cast, which is abnormal.

[0111] Therefore, the black level correction unit is used to correct the dark signal output by the image sensor to eliminate the influence of the dark signal and improve the image quality.

[0112] 2. Automatic white balance unit

[0113] Because the sensor cannot change its photosensitivity according to changes in ambient light like the human eye, an additional unit is needed to simulate the photosensitivity of the human eye and restore the white under different ambient light to true white.

[0114] Specifically, the automatic white balance unit is used to restore the white image formed under ambient light of different color temperatures to true white (usually the white observed by the human eye under natural daylight ambient light).

[0115] 3. Integrated processing unit

[0116] An integrated processing unit is a processing unit that integrates multiple functions, such as Figure 1In the example, the integrated processing unit is a processing unit that integrates noise reduction function, demosaic function and HDR (High Dynamic Range) function.

[0117] In the actual implementation process, the specific functions integrated in the integrated processing unit can be selected and set according to actual needs. Figure 1 Different from the above, the black level correction unit and the automatic white balance unit can be integrated into the integrated processing unit, and the noise reduction function, the demosaicing function and the HDR function can be respectively used as three independent processing units. This embodiment does not specifically limit which functions are integrated into the integrated processing unit and which processing units are separately set in the image processing model.

[0118] 4. Lens shading correction unit

[0119] The lens shading correction unit is used to correct the lens shading phenomenon. Lens shading is the phenomenon of uneven image brightness caused by the optical characteristics of the lens and image sensor. Usually, the center of the image is brighter, while the edge is darker. The purpose of the LSC unit is to correct this brightness unevenness so that the image has the same brightness level in the entire field of view.

[0120] Figure 1 Some processing units that may be included in the image processing model are introduced in the example. In the actual implementation process, the image processing model may also include Figure 1 Processing units other than those introduced above, such as bad point correction (BPC), color correction matrix (CCM), gamma correction, CSC color space conversion, edge enhancement, etc. This embodiment does not limit the processing units included in the image processing model and the specific execution order between the processing units, which can be selected and set according to actual needs.

[0121] Based on the image processing model introduced above, it is necessary to deploy the image processing model in the terminal device in an application for realizing the image shooting function, and then the image processing model can play the role of image processing accordingly.

[0122] Ideally, if the model can be fully deployed in the terminal device and all functions in the model can work well, the hue information of the image output by the image processing model should be consistent with the hue information of the original image. In one implementation, the original image can be, for example, a RAW image, because the hue information of a RAW image is generally more accurate, so ideally, the hue of the image finally output by the model can be closer to the real scene.

[0123] However, during the model deployment process, the model accuracy will inevitably be lost due to reasons such as model quantization and loss of floating-point precision.

[0124] And with the continuous development of model technology, complex models that are compatible with multiple tasks and have high integration have become more and more mainstream in recent years. For example, the image processing model introduced above has an integrated processing unit, in which a variety of image processing functions are integrated. For models with high complexity and strong compatibility, when the integrated processing unit is compatible with multiple functions, it is inevitable that the processing effects of the compatible functions cannot reach the extreme, so some effects will be difficult to control.

[0125] Due to the above two reasons, the image output by the image processing model may have color cast problems, that is, the color of the final output image deviates from the color in the actual scene. If it is quantified, it means that the R / G and B / G ratios of the final output image have changed compared to the original image, resulting in a color shift in the final output image.

[0126] In terms of effect, the color cast problem will cause the overall tone of the image to lean towards a certain color. For example, the image may appear reddish, greenish, or bluish, etc.

[0127] If the color cast problem is solved by model iteration, the model may need to be iterated repeatedly, which will make the solution to the color cast problem more complicated and time-consuming, and the training effect is difficult to control.

[0128] The following technical concept is proposed in the present application: when inputting data into an image processing model, in addition to inputting the original image, an offset parameter of at least one color channel is also input at the same time, so that during the processing, the image processing model performs an offset processing on at least one color channel of the image according to the offset parameter of at least one color channel. The offset processing here can be understood as an offset in the opposite direction of the color cast problem of the image, thereby specifically solving the problem of image color cast.

[0129] The technical solution provided in this application can be applied to terminal devices. The following first briefly introduces the terminal devices.

[0130] For example, Figure 2 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application.

[0131] The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0132] The processor 110 may include one or more processing units. Different processing units may be independent devices or integrated into one or more processors. The processor 110 may also be provided with a memory for storing instructions and data.

[0133] The terminal device realizes the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering.

[0134] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the terminal device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0135] The terminal device can realize camera functions such as shooting and recording through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0136] The camera 193 is used to capture static images or videos. In some embodiments, the terminal device may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0137] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0138] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device. In other embodiments of the present application, the terminal device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0139] The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes a layered architecture system as an example to exemplify the software structure of the terminal device.

[0140] For example, Figure 3 A schematic diagram of the software structure of a terminal device provided in an embodiment of the present application.

[0141] like Figure 3 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through interfaces. In some embodiments, the system may include an application layer, an application framework layer, a hardware abstraction layer (HAL), a driver layer, and a hardware layer.

[0142] It should be noted that the embodiments of the present application are illustrated using the Android system as an example. In other operating systems (such as Hongmeng system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those of the embodiments of the present application, the solutions of the present application can also be implemented.

[0143] The application layer can include a series of application packages. Figure 3 As shown, the application package may include applications such as a camera and a gallery. The camera application is, for example, an application for implementing the image capture function introduced in this application. And after the image capture is completed, the user can view the captured image in the gallery application.

[0144] The application layer may also include calendar, phone, map, navigation, email, social, wireless local area network (WLAN), Bluetooth, music, video, short message, lock screen application, setting application. Of course, the application layer may also include other application packages, such as payment application, shopping application, banking application, social application and other third-party applications, which are not limited in this application.

[0145] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the application programs in the application layer. Figure 3 As shown, the application framework layer may include a camera access interface. The camera access interface may include camera management and camera equipment. Among them, camera management may be used to provide an access interface for managing cameras, and camera equipment may be used to provide an interface for accessing cameras.

[0146] Furthermore, the application framework layer may also include some predefined functions. For example, it may include an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, and a camera server unit, etc., and the present embodiment of the application does not impose any restrictions on this. Among them, the camera server unit of the application framework layer can be started during the startup phase of the terminal device and can be used to transmit and save relevant information of the camera.

[0147] And, the hardware abstraction layer is used to abstract the hardware, implement the encapsulation of the Linux kernel driver, provide an interface to the upper layer, and shield the implementation details of the low-level hardware. For example, the hardware abstraction layer may include Figure 3 The camera hardware abstraction layer, other hardware device abstraction layers, and camera algorithm library shown.

[0148] The camera hardware abstraction layer may include camera device 1, camera device 2, etc. The camera hardware abstraction layer may be connected to the camera algorithm library, and the camera hardware abstraction layer may call the algorithm in the camera algorithm library.

[0149] The camera algorithm library may include algorithm instructions such as camera algorithms and image algorithms and perform some image processing steps. Exemplarily, the camera algorithm library may include an image processing model, so as to implement the image processing described in this application.

[0150] And, the driver layer is used to provide drivers for different hardware devices. For example, the driver layer can include camera drivers, digital signal processor drivers, and graphics processor drivers. The camera driver is the driver layer of the camera device, which is mainly responsible for the interaction with the hardware module.

[0151] Furthermore, the hardware layer may include sensors, image processors, digital signal processors, graphics processors, memory and other hardware devices, which are used to interact with the camera driver to achieve the purpose of image capture.

[0152] In the process of photographing, for example, the camera application of the application layer can respond to user operations and send image acquisition instructions to the camera device in the camera access interface, where image acquisition includes image acquisition in the preview stage and image acquisition in the shooting stage. After that, the camera device sends relevant instructions to the camera HAL, and the camera HAL sends relevant instructions to the camera device driver, and the camera device driver interacts with the relevant sensors and processors of the hardware layer to complete the purpose of image shooting.

[0153] After the image is captured, the captured image is displayed in the camera application or gallery application through the reverse data flow as described above. In the reverse data flow, the captured image can be further processed in some hierarchical structures.

[0154] For example, the HAL layer may be configured to further process the captured image. For example, the HAL layer may call an image processing model in a camera algorithm library to process the image so that the final output image has a good performance.

[0155] Based on the above introduction, the technical solution provided by this application is introduced in detail below.

[0156] In order to enable the image processing model to have the ability to shift the hue of an image according to the input hue shift coefficient, the image processing model must first be trained accordingly during the model training phase.

[0157] Let's first combine Figure 4 This paper introduces the training process of image processing models. Figure 4 A schematic diagram of the training process of the image processing model provided in an embodiment of the present application.

[0158] like Figure 4 As shown, the training process of the image processing model may include the following steps:

[0159] 1. Determine a sample offset factor for at least one target color channel.

[0160] In this embodiment, at least one target color channel may be predetermined among the R color channel, the B color channel, and the G color channel. Based on the above-introduced grayscale world method and the related concepts of AWB gain, it can be understood that it is usually necessary to fix a certain color channel as a reference, and then perform offset processing on the remaining two color channels, so two color channels may be selected from the three color channels of R\G\B as the target color channels.

[0161] Because the human eye is usually more sensitive to green, the G color channel can be fixed, and the R color channel and the B color channel can be selected as the target color channel. Alternatively, the G and B color channels can be determined as the target color channels, or the G and R color channels can be determined as the target color channels. In this embodiment, the specific selection of the target color channel is not limited. The following is described by taking the target color channels as the R channel and the B channel as an example, and the rest of the situations are similar, so they are not repeated.

[0162] And in the training phase, in this embodiment, a sample shift coefficient of at least one target color channel is determined. Exemplarily, a sample shift coefficient is determined for the R color channel, and a sample shift coefficient is determined for the B color channel.

[0163] exist Figure 4 In the example of , it is assumed that the sample offset coefficient set for the R color channel is 1.25, and the sample offset coefficient set for the B color channel is 1.

[0164] In this embodiment, the sample offset coefficient is used to indicate to offset the tone parameter corresponding to the target color channel, wherein the larger the sample offset coefficient is, the greater the degree of offset of the tone parameter is.

[0165] The hue parameter is used to indicate the hue information in the image. In one implementation, the AWB gain of the image can be used as the hue parameter of the image. The AWB gain includes the gain parameter (R_gain) of the R color channel, the gain parameter (G_gain) of the G color channel, and the gain parameter (B_gain) of the B color channel.

[0166] In another implementation, the average value Rmean of the R color channel, the average value Gmean of the G color channel, and the average value Bmean of the B color channel may also be determined for the image. Then, Rmean / Gmean, Bmean / Gmean, and Gmean are used as the hue parameters of the image. It is understandable that this implementation is essentially the same as AWB gain, but the expression is slightly different.

[0167] Here, a possible implementation method for determining the sample shift coefficient is described.

[0168] In one implementation, in this embodiment, some color cast scenes may be constructed, and then the sample shift coefficients may be determined based on these color cast scenes. Constructing the color cast scenes means adjusting the hue parameters of the sample image to cause the image to have a color cast phenomenon.

[0169] For example, for any target color channel, first, among the gain parameters of at least one color channel of the sample image, the gain parameter of the target color channel may be determined as the first initial gain parameter.

[0170] Based on the above introduction, it can be determined that the gain parameters of at least one color channel of the sample image may include R_gain, G_gain, and B_gain. Assuming that the R channel and the B channel are target color channels, then among R_gain, G_gain, and B_gain of the sample image, R_gain may be determined as the first initial gain parameter of the R channel, and B_gain may be determined as the first initial gain parameter of the B channel.

[0171] Afterwards, the first initial gain parameter may be numerically adjusted to obtain a first adjusted gain parameter of the target color channel.

[0172] The numerical value adjustment process may be any one of increasing the numerical value, decreasing the numerical value, and keeping the numerical value unchanged. In this embodiment, when generating the sample offset coefficient, one of the multiple numerical value adjustment processes described above may be randomly selected for any target color channel. This implementation method is to randomly construct a color cast scene to determine the sample offset coefficient.

[0173] Alternatively, a fixed type of numerical adjustment processing can be set for the target color channel. For example, after preliminary analysis, it can be determined that the image processing model will present a greenish color cast effect. In order to simulate this color cast effect, for example, the numerical adjustment processing can be fixedly set for the R color channel to reduce the value. This implementation method is to construct a color cast scene based on the color cast characteristics of the image processing model, thereby constructing a sample offset coefficient.

[0174] In addition to determining what kind of numerical adjustment to perform, it is also necessary to determine the degree of numerical adjustment. Because R_gain, G_gain and B_gain are usually within a certain numerical range, if R_gain, G_gain and B_gain are adjusted too large or too small, the constructed color cast scene may be unreasonable.

[0175] Therefore, in one implementation, in this embodiment, for example, a first value may be generated within a preset value range, and then the first initial gain parameter may be numerically adjusted according to the first value, wherein the preset value range may be selected according to actual needs.

[0176] Exemplarily, it is assumed that the tone parameters of the sample image are (2.0, 1.0, 1.5), where R_gain is 2, G_gain is 1, and B_gain is 1.5.

[0177] And assuming that for the R color channel, a first value of 0.1 is generated within a preset value range, and assuming that the selected value adjustment process is to reduce the value, a value adjustment process can be performed on R_gain(2) of the sample image, so that the first adjustment gain parameter of the R color channel after adjustment is 1.6.

[0178] Also, assuming that for the B color channel, a first value of 0.1 is generated within a preset value range, but assuming that the selected numerical adjustment processing is to keep the value unchanged, a numerical adjustment processing can be performed on B_gain(1.5) of the sample image, so that the first adjustment gain parameter of the B color channel after adjustment is 1.5.

[0179] The above introduces the implementation of constructing a color cast scene, and the sample offset coefficient in this embodiment is used to offset the hue to correct the color cast phenomenon. Therefore, for example, the ratio of the first initial gain parameter and the first adjustment gain parameter can be determined as the sample offset coefficient of the target color channel.

[0180] For instance, in the above example, for the R channel, the ratio result of 2 / 1.6, 1.25, can be determined as the sample offset coefficient of the R color channel to indicate that the hue parameter of the R color channel needs to be increased to the extent indicated by 1.25 in order to compensate for the color cast of R_gain from 2 to 1.6.

[0181] And, for the B channel, the sample offset coefficient can be determined to be 1.

[0182] The implementation method introduced above is to construct a virtual color cast scene by numerically adjusting the hue parameter based on the sample image to determine the sample shift coefficient. This implementation method can ensure the rationality and practicality of the determined sample shift coefficient.

[0183] Alternatively, in another implementation, the operator may directly input the sample offset coefficient based on experience. For example, the operator directly generates a sample offset coefficient of 1.25 for the R channel and a sample offset coefficient of 1 for the B channel. This may also achieve the above purpose, but this implementation relies on manual experience.

[0184] Also, usually the sample images obtained as training data are high-definition images. In order to simulate the images actually collected in the actual scene, noise can be first superimposed on the sample images to perform data degradation. Then, the color cast scene introduced above is constructed based on the sample images superimposed with noise to determine the sample shift coefficient of at least one target data channel.

[0185] 2. Input the sample image into the image processing model to obtain a first processed image output by the image processing model, and determine the tone parameters of the first processed image.

[0186] During the model training phase, sample images may be provided. To ensure the learning effect of the image processing model, the sample images may be RAW images, for example.

[0187] In order to clarify the image processing effect of the image processing model itself, refer to Figure 4 In this embodiment, the sample image can be input into the image processing model to obtain a first processed image output by the image processing model. This process can be understood as a step that a traditional image processing model would perform. Referring to the above description, it can be understood that the first processed image may have a color cast problem.

[0188] And in order to measure the hue of the first processed image, in this embodiment, the hue parameter of the first processed image can be determined. Figure 4 , assuming that the tone parameters of the first processed image are (1.6, 1.0, 1.5), where R_gain is 1.6, G_gain is 1, and B_gain is 1.5.

[0189] And reference Figure 4 It can also be determined that the hue parameters of the sample image are (2.0, 1.0, 1.5), where R_gain is 2, G_gain is 1, and B_gain is 1.5. By comparing the hue parameters of the two images, it can be understood that after being processed by the image processing model, the first processed image has a color cast problem compared to the sample image, specifically, the R color channel is weakened and / or the G color channel is enhanced, for example, the first processed image shows a greenish effect.

[0190] 3. Perform an offset process on the tone parameters of the first processed image according to the sample offset coefficient to obtain a reference tone parameter.

[0191] In this embodiment, the purpose of setting the sample offset coefficient is to allow the image processing model to use it as a basis, and to offset the hue parameter of the corresponding target color channel according to the sample offset coefficient during the image processing process.

[0192] In the model training stage, a parameter needs to be determined to measure whether the hue parameter of the image finally output by the image processing model has been effectively processed. In other words, the expected model output result should be determined first. This step can also be understood as determining the labeled data in the image training process.

[0193] In one implementation, the tone parameter of the first processed image may be shifted according to the sample shift coefficient, so as to obtain the reference tone parameter. Based on the above-described method for determining the sample shift coefficient, the product of the sample shift coefficient and the corresponding tone parameter may be determined as the reference tone parameter.

[0194] Therefore, for example, for any target color channel, first, the gain parameter of the target color channel can be determined as the second initial gain parameter among the gain parameters of at least one color channel of the first processed image.

[0195] Reference Figure 4 , among the tone parameters (1.6, 1.0, 1.5) of the first processed image, R_gain (1.6) can be determined as the second initial gain parameter of the R channel, and B_gain (1.5) can be determined as the second initial gain parameter of the B channel.

[0196] Then, the second adjustment gain parameter of the target color channel may be determined according to the product of the sample offset coefficient of the target color channel and the second initial gain parameter.

[0197] For example, refer to Figure 4 , assuming that the sample offset coefficient of the R color channel is 1.25, and the sample offset coefficient of the B color channel is 1. Then for the R color channel, it can be determined that the adjusted second adjustment gain parameter is the product of 1.25 and 1.6, that is, the second adjustment gain parameter of the R color channel is 2.

[0198] And, for the B color channel, it can be determined that the adjusted second adjustment gain parameter is the product of 1 and 1.5, that is, the second adjustment gain parameter of the B color channel is 1.5.

[0199] Afterwards, among the gain parameters of at least one color channel of the first processed image, the gain parameter of the target color channel may be replaced with the corresponding second adjustment gain parameter to obtain a reference hue parameter.

[0200] Reference Figure 4 , after making corresponding substitutions based on the tone parameters (1.6, 1.0, 1.5) of the first processed image, we can get Figure 4 Reference hue parameters shown (2.0, 1.0, 1.5).

[0201] 4. Input the sample image and the sample shift coefficient of at least one target color channel into the image processing model to obtain a second processed image output by the image processing model, and determine the actual tone parameter of the second processed image.

[0202] After preparing the training data introduced above, namely the sample image, sample offset coefficient and reference tone parameter, the training process of the image processing model can be executed.

[0203] During the training process, refer to Figure 4 , the sample image and the sample offset coefficient of at least one target color channel can be input into the image processing model, and then the image processing model can output a second processed image after completing the corresponding image processing.

[0204] During the model training phase, we expect the image processing model to adjust the hue parameters of the corresponding target color channels to a certain extent based on the sample offset coefficient, and the reference hue parameters introduced above are the expected adjusted results.

[0205] The processing result actually output by the current image processing model is the second processed image. In order to clarify the processing effect of the image processing model, the tone parameters of the second processed image can be determined in this embodiment, and the tone parameters of the second processed image can be determined as (or called) actual tone parameters.

[0206] like Figure 4 As shown, the actual tone parameters of the second processed image are, for example, (2.0, 1.0, 1.5).

[0207] 5. Update the model parameters of the image processing model according to the reference tone parameters and the actual tone parameters.

[0208] In this embodiment, the reference tone parameter can be understood as the correct result marked in advance, and the actual tone parameter can be the actual result output by the model training result. Our desired goal is that the actual tone parameter of the second processed image output by the model is as consistent as possible with the reference tone parameter we expect.

[0209] Therefore, in this embodiment, the loss value can be determined according to the reference tone parameter and the actual tone parameter, and the model parameters of the image processing model can be updated according to the loss value to achieve optimization and update of the model.

[0210] Based on the model training process introduced above, the image processing model can learn the following capabilities: in the image processing process, the hue parameters of the corresponding color channels are offset according to the input sample offset parameters. When the image processing model has such capabilities, the hue offset coefficient can be set according to the actual color cast during the model application stage, so that the image processing model can perform hue offset on the image according to the hue offset coefficient, thereby correcting the color cast.

[0211] On this basis, the following Figure 5 The image processing method provided in this application is introduced in detail. Figure 5 A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 6 A schematic diagram of an implementation of determining a hue shift coefficient provided in an embodiment of the present application.

[0212] like Figure 5 As shown, the original image captured by the camera and the hue shift coefficient of at least one target color channel can be determined.

[0213] The original image may be, for example, a RAW image, and the target color channel is similar to that described above, for example, the R color channel and the B color channel may be set as the target color channel. The hue shift coefficient in this embodiment is similar to the sample shift coefficient described above, and is used to indicate that the hue parameter corresponding to the target color channel is shifted.

[0214] Refer to later Figure 5 , inputting the original image and the hue shift coefficient of at least one target color channel into the image processing model, thereby obtaining the target image output by the image processing model.

[0215] Among them, during the image processing process, the image processing model will offset the hue parameters corresponding to the target color channel according to the hue shift coefficient of at least one target color channel to achieve compensation for the image color cast, thereby ensuring that the hue parameters of the output target image are as consistent as possible with the hue parameters of the original image, so as to solve the image color cast problem in a targeted manner.

[0216] In this embodiment, the hue shift coefficient of the target color channel is used to indicate the shift degree of the hue parameter of the target color channel, so the determination of the number of hue shifts of the target color channel is also very important.

[0217] In one implementation, a color cast test may be performed on the image processing model to determine the hue shift coefficient of the image processing model.

[0218] like Figure 6As shown, the test input image can be input into the image processing model to obtain the test output image output by the image processing model. Then, the hue shift coefficient of at least one target color channel can be determined according to the hue parameter of the test input image and the hue parameter of the test output image.

[0219] For example, for any target color channel, the ratio of the hue parameter of the target color channel in the test input image to the hue parameter of the target color channel in the test output image is determined as the hue shift coefficient of the target color channel.

[0220] For example, Figure 6 In the example, assuming that the hue parameters of the test input image are (2.0, 1.0, 1.5), and the hue parameters of the test output image are (1.6, 1.0, 1.5), then it can be determined that the hue shift coefficient of the R channel is 1.25 (2 / 1.6), and the hue shift coefficient of the B channel is 1.5 (1.5 / 1).

[0221] Furthermore, for images captured in different shooting environments, the color cast effects presented by the image processing model may be different. Therefore, in order to ensure the correctness of the color cast correction processing, for example, test input images can be captured in a variety of different image shooting environments, so as to determine the corresponding hue shift coefficients for each of the various shooting environment parameters.

[0222] Exemplarily, for example, a test input image may be captured when the ambient brightness (lux) is 1000, and then the above-described process may be performed to determine the hue shift coefficient of the R channel and the hue shift coefficient of the B channel when the ambient brightness is 1000. Furthermore, a test input image may be captured when the ambient brightness (lux) is 100, and then the above-described process may be performed to determine the hue shift coefficient of the R channel and the hue shift coefficient of the B channel when the ambient brightness is 1000. In the actual implementation process, the specific ambient brightness at which the hue shift coefficient is determined may be set according to actual needs.

[0223] Similarly, different sensitivities (ISO) may be set in the image acquisition scene, and then test images may be acquired at different sensitivities, and then the above-described process may be performed to determine the hue shift coefficient of the R channel and the hue shift coefficient of the B channel at different sensitivities.

[0224] In a possible implementation, in the stage of determining the hue shift coefficient, multiple combinations of different ambient brightness and different sensitivities may be determined, and then test input images may be collected for the shooting scenes corresponding to the multiple combinations, and the hue shift coefficients corresponding to the various combinations may be determined based on the above-described process. Then, for example, a first curve of the hue shift coefficient corresponding to the R channel and a second curve of the hue shift coefficient corresponding to the B channel may be fitted for the hue shift coefficients corresponding to the various combinations.

[0225] In the first curve, for example, the horizontal axis is the ambient brightness, the vertical axis is the sensitivity (or vice versa), and then the curve is the hue shift coefficient of the R channel corresponding to the combination of the ambient brightness and the sensitivity. The second curve is similar.

[0226] For this implementation method of determining the first curve and the second curve, during the actual implementation process, the hue shift coefficient of the R channel can be determined in the first curve, and the hue shift coefficient of the B channel can be determined in the second curve according to the ambient brightness and sensitivity of the original image captured, thereby ensuring that the determined hue shift coefficient is adapted to the color cast effect corresponding to the current image capture environment, thereby ensuring the correctness of the color cast correction based on the hue shift coefficient.

[0227] based on Figure 6 In the example shown, it is assumed that the hue shift coefficient of the R channel is 1.25, and the hue shift coefficient of the B channel is 1.5. At the same time, it is assumed that the hue parameters of the original image are (2.5, 1.0, 1.5). If the image processing model does not perform color cast correction, the hue parameters of the output image after image processing may be (2.0, 1.0, 1.0).

[0228] Because the image processing model has gone through the training process described above, the color cast correction indicated by the hue shift coefficient is applied, so the hue parameters of the target image output by the image processing model may be (2.5, 1.0, 1.5), or at least close to this result. It can be understood that the ultimate ideal result is that the hue parameters of the target image output by the image processing model are consistent with those of the sample image, thereby solving the color cast problem.

[0229] In this embodiment, by adding a hue shift coefficient to the input data to control the color tendency of the output image, it is ensured that the target image finally output by the image processing model can correctly represent the hue in the actual environment. In the specific processing process, by controlling the image processing model to refer to the hue shift coefficient, the hue parameters of the image are shifted to a corresponding degree during the image processing process, thereby achieving targeted correction of the color cast processing of the image processing model.

[0230] The above embodiment introduces that the image processing model combines the hue shift coefficient to shift the hue of the entire image, so as to achieve the purpose of solving the color cast problem of the image as a whole. However, in the actual implementation process, in addition to the color cast problem of the entire image, the image may also have a local color cast problem, that is, a certain area in the image may have a color cast problem.

[0231] Therefore, in the technical solution of this application, Figure 1 Based on the image processing model introduced, a local color correction unit can be further added.

[0232] For example, you can refer to Figure 7 To understand, Figure 7 Schematic diagram of the image processing model provided in the embodiment of the present application Figure 2 .

[0233] Figure 7 Most of the processing units in the Figure 1 The introductions are similar, except that, in this embodiment, a local color correction unit may be added to the image processing model, wherein the local color correction unit is used to implement color correction processing on a local part of the image.

[0234] In one implementation, the local color correction unit can be set after the integrated processing unit. Because the problems with the integrated processing unit introduced above may cause the intermediate image output after processing by the integrated processing unit to have a more serious color cast problem, setting the local color correction unit after the integrated processing unit can ensure the targetedness and comprehensiveness of the local correction processing.

[0235] Combine the following Figures 8 to 9 The specific processing process of the local color correction unit is introduced in detail. Figure 8 A schematic diagram of the processing process of local color cast correction provided in an embodiment of the present application, Fig. 9 A schematic diagram of determining the second compensation matrix provided in an embodiment of the present application.

[0236] In one implementation, an intermediate image output by the integrated processing unit may be obtained during the processing of the image processing model.

[0237] It should be noted that during the processing of the image processing model, the RAW image will be split into RGB channels, and the image split into RBG channels is 1 / 4 the size of the original image. And, the integrated processing unit will then process the image split into RGB channels, so the intermediate image output by the integrated processing unit is also RGB channels, which is also 1 / 4 the size of the original image.

[0238] Afterwards, n image regions are divided in the original image, and n image regions are also divided in the intermediate image, wherein the n image regions in the original image and the n image regions in the intermediate image correspond one to one, and n is an integer greater than or equal to 1.

[0239] For example, you can refer to Figure 8 Understand, assuming that the original image is divided into 16 regions, and the intermediate image is also divided into 16 regions, wherein the size and number of regions divided in the original image and the intermediate image are consistent, so it can be ensured that the 16 regions in the original image and the 16 regions in the intermediate image are one-to-one corresponding.

[0240] Then, for any image region, a regional compensation value of at least one target color channel corresponding to the image region can be determined based on the first pixel value information of the image region in the original image and the second pixel value information of the image region in the intermediate image.

[0241] The pixel value information may include, for example, the average pixel value of the first color channel of each pixel point in the image area, the average pixel value of the second color channel of each pixel point in the image area, and the average pixel value of the third color channel of each pixel point in the image area.

[0242] In a possible implementation manner, the first color channel, the second color channel, and the third color channel may be an R color channel, a B color channel, and a G color channel, respectively.

[0243] Reference Figure 8 , taking the first image area in the upper left corner as an example, the first pixel value information of this image area in the original image includes as well as in, It is the average value of the R pixel values ​​of each pixel in the image area of ​​the original image. It is the average value of the B pixel values ​​of each pixel in the image area of ​​the original image. It is the average value of the G pixel values ​​of each pixel point in the image area of ​​the original image.

[0244] And, the second pixel value information of the image area in the intermediate image includes as well as in, It is the average value of the R pixel values ​​of each pixel in the image area of ​​the intermediate image. It is the average value of the B pixel values ​​of each pixel in the image area of ​​the intermediate image. It is the average value of the G pixel values ​​of each pixel point in the image area of ​​the intermediate image.

[0245] Afterwards, the ratio of the average pixel value of the first color channel to the average pixel value of the second color channel in the first pixel value information can be determined as the first ratio, and the ratio of the average pixel value of the first color channel to the average pixel value of the second color channel in the second pixel value information can be determined as the second ratio. Finally, the ratio of the first ratio to the second ratio is determined as the regional compensation value of the target color channel.

[0246] For example, when the first color channel is the R channel and the second color channel is the G channel, the above content means that the first pixel value information is and The ratio of is determined as the first ratio, that is, Figure 8 As shown in And the second pixel value information and The ratio of is determined as the second ratio, that is, Figure 8 As shown in

[0247] It is understandable that the The meaning of R_gain is similar to that of R_gain mentioned above. However, for global images, the AWB gain can be directly obtained to determine R_gain, but for local images, the AWB gain cannot be directly obtained, so it needs to be calculated through the above calculation process. as well as

[0248] in, In fact, it is the gain parameter of the R color channel of the image area in the original image, and In fact, it is the gain parameter of the R color channel of the image area in the intermediate image, and its change reflects the color cast of the R channel and / or the G channel. Therefore, the ratio of the first ratio and the second ratio can be determined, and this ratio is the regional compensation value of the R channel, where the meaning of the regional compensation value is how much the R color channel in the area should be offset to compensate for the color cast in the area.

[0249] For example, Figure 8 In the example, suppose is 1.07, and assuming is 1.08, then it can be understood that a color cast effect of red enhancement and / or green reduction occurs in this area, and then the ratio result of 1.07 / 1.08 can be determined as the regional compensation value of the R color channel.

[0250] Alternatively, when the first color channel is the B channel and the second color channel is the G channel, the above content means that the first pixel value information is and The ratio of is determined as the first ratio, that is, Figure 8 As shown in And the second pixel value information and The ratio of is determined as the second ratio, that is, Figure 8 As shown in

[0251] It is understandable that the The meaning of B_gain is similar to that of B_gain mentioned above. However, for global images, the AWB gain can be directly obtained to determine B_gain, but for local images, the AWB gain cannot be directly obtained, so it needs to be calculated through the above calculation process. as well as

[0252] in, In fact, it is the gain parameter of the B color channel of the image area in the original image, and In fact, it is the gain parameter of the B color channel of the image area in the intermediate image, and its change reflects the color cast of the B channel and / or the G channel. Therefore, the ratio of the first ratio and the second ratio can be determined, and this ratio is the regional compensation value of the B channel, where the meaning of the regional compensation value is how much the B color channel in the area should be offset to compensate for the color cast in the area.

[0253] For example, Figure 8 In the example, suppose is 0.87, and assuming is 0.84, then it can be understood that a color cast effect of weakening blue and / or enhancing green occurs in this area, and then the ratio result of 0.87 / 0.84 can be determined as the regional compensation value of the B color channel.

[0254] After determining the area compensation values ​​corresponding to the respective target color channels, the area compensation values ​​of the respective image regions may be set as the pixel compensation values ​​corresponding to the respective pixel points in the respective image regions for any target color channel.

[0255] For example, for a certain image area, assuming that the area compensation value of its R channel is a, and assuming that there are 20 pixels in the image area, then the pixel compensation values ​​of the R channels of these 20 pixels are all set to a. Similar operations are performed for each image area, and then the corresponding pixel compensation value can be set for each pixel.

[0256] After the corresponding pixel compensation value is set for each pixel point in the intermediate image, a matrix composed of the pixel compensation values ​​corresponding to each pixel point in the intermediate image can be used to determine a first compensation matrix.

[0257] Exemplarily, assuming that the intermediate image contains 40×80 pixels, a matrix is ​​determined according to the pixel compensation value corresponding to each pixel, and a first compensation matrix with a size of 40×80 can be obtained.

[0258] It is understandable that the purpose of determining the first compensation matrix is ​​to subsequently perform targeted color cast correction on each image area. However, because each image area is processed separately in the above processing, after the local color cast correction is performed, the final performance of the connection between the image areas may not be good.

[0259] Therefore, further, for a transition region between any two adjacent image regions, the pixel compensation values ​​of the pixels in the transition region may be smoothed, thereby obtaining smoothed pixel compensation values ​​of the pixels in the transition region.

[0260] Then, the smoothed pixel compensation values ​​of the pixels in the transition area are used to replace the pixel compensation values ​​of the pixels at the corresponding positions in the first compensation matrix, thereby updating the first compensation matrix.

[0261] For example, a region of a preset shape and a preset size adjacent to two adjacent regions can be determined as a transition region. For example, the transition region can be set to occupy the same area in the two adjacent regions. Fig.10 , where image region 6 and image region 10 are two adjacent regions, then for example, a rectangular region of a fixed size of a×b at the adjacent position of the two adjacent regions can be determined as a transition region, and the area occupied by this rectangular region in the two image regions is the same.

[0262] In the actual implementation process, the specific setting method of the transition area can be selected according to actual needs. For example, it can be set to an ellipse, etc., as long as the transition area is set at the adjacent position of two adjacent areas.

[0263] Furthermore, when performing the smoothing process, for example, Gaussian smoothing may be performed according to the pixel compensation values ​​in the transition region, thereby determining the smoothed pixel compensation values ​​corresponding to each pixel point in the transition region.

[0264] Based on the smoothing process for the transition area introduced above, it can be ensured that when the color cast correction process is performed on a part of the image, the color cast correction effect between the image areas can be smoothly transitioned, thereby avoiding the problem of discontinuous tones in the image.

[0265] After determining the first compensation matrix introduced above, for example, compensation processing can be performed directly on the intermediate image according to the first compensation matrix.

[0266] Exemplarily, because the sizes of the first compensation matrix and the intermediate image are equal, the matrix positions in the first compensation matrix and the pixel positions in the intermediate image also correspond one to one. Therefore, the first compensation matrix of the R channel can be multiplied with the R pixel value of each pixel in the intermediate image at the corresponding positions, and the first compensation matrix of the B channel can be multiplied with the B pixel value of each pixel in the intermediate image at the corresponding positions, thereby achieving color cast compensation for the intermediate image.

[0267] Furthermore, considering the serious attenuation of the four corners of the lens, it is also possible to consider increasing the hue compensation ratio of the four corners of the image on the basis of the first compensation matrix.

[0268] In one implementation, for example, a second compensation matrix can be pre-set, wherein the matrix size of the second compensation matrix is ​​also consistent with the image size of the intermediate image, and the compensation value of each matrix position in the second compensation matrix is ​​proportional to the first distance corresponding to the matrix position, wherein the first distance is the distance between the matrix position and the center position of the second compensation matrix.

[0269] That is to say, the closer the matrix position is to the center of the second compensation matrix, the smaller the compensation value is, and the farther the matrix position is from the center of the second compensation matrix, the larger the corresponding compensation value is. This can effectively achieve the effect that the compensation values ​​of the four corners of the image are greater than the compensation values ​​of the central area of ​​the image, so as to increase the compensation ratio of the four corners of the image.

[0270] For example, the second compensation matrix can be set in the following manner: Fig. 9 For example, based on the image size of the intermediate image, a plurality of different compensation areas may be divided, wherein the compensation value in the compensation area 901 may be set to Fig. 9 The parameter 1 shown, and the compensation value in the compensation area 902 can be set to, for example, Fig. 9 As shown in parameter 2, the compensation value in compensation area 903 can be set to Fig. 9 As shown in parameter 3, the compensation value in the compensation area 904 can be set to Fig. 9 Parameter 4 is shown. It should be noted that Fig. 9 The four compensation areas in are non-intersecting.

[0271] For example, parameter 4>parameter 3>parameter 2>parameter 1 may be set, so that the closer to the four corners of the image, the larger the compensation value, and the closer to the center of the image, the smaller the compensation value.

[0272] In the actual implementation process, the second compensation matrix is ​​pre-set, so the specific compensation values ​​in the second compensation matrix can be set according to actual needs, as long as the above-mentioned effect of the closer to the four corners of the image, the larger the corresponding compensation value, and the closer to the center of the image, the smaller the corresponding compensation value can be met.

[0273] On the basis of the second compensation matrix introduced above, for example, the first compensation matrix and the second compensation matrix can be multiplied at corresponding positions to achieve the effect of increasing the four-corner compensation ratio indicated by the second compensation matrix, and superimposed on the first compensation matrix to obtain the target compensation matrix.

[0274] Then, the intermediate image can be partially compensated according to the target compensation matrix. Similarly, the target compensation matrix of the R channel can be multiplied with the R pixel value of each pixel in the intermediate image at the corresponding position, and the target compensation matrix of the B channel can be multiplied with the B pixel value of each pixel in the intermediate image at the corresponding position, so as to realize the color cast compensation of the intermediate image.

[0275] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.

[0276] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0277] The model training method and image processing method of the embodiment of the present application have been described above, and the device for executing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above model training method and image processing method.

[0278] The model training method and image processing method provided in the embodiment of the present application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can refer to the above-mentioned related descriptions, which will not be repeated here.

[0279] In one implementation, an embodiment of the present application provides an electronic device, Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0280] like Fig.10 As shown, the electronic device 100 includes: a processor 1001 and a memory 1002; the memory 1002 stores computer-executable instructions; the processor 1001 executes the computer-executable instructions stored in the memory 1002, so that the electronic device 100 executes the above method.

[0281] When the memory 1002 is independently provided, the electronic device further includes a bus 1003 for connecting the memory 1002 and the processor 1001 .

[0282] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0283] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0284] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0285] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0286] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0287] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A model training method, characterized in that: include: Determine a sample shift coefficient of at least one target color channel, where the sample shift coefficient is used to indicate that a hue parameter corresponding to the target color channel is shifted; Inputting the sample image into the image processing model to obtain a first processed image output by the image processing model, and determining a tone parameter of the first processed image; performing an offset process on the tone parameter of the first processed image according to the sample offset coefficient to obtain a reference tone parameter; Inputting the sample image and the sample shift coefficient of the at least one target color channel into an image processing model to obtain a second processed image output by the image processing model, and determining an actual hue parameter of the second processed image; The model parameters of the image processing model are updated according to the reference tone parameter and the actual tone parameter.

2. The method according to claim 1, characterized in that The color tone parameter is an automatic white balance (AWB) gain, and the AWB gain includes a gain parameter of at least one color channel; The determining of the sample offset coefficient of at least one target color channel comprises: For any of the target color channels, among the gain parameters of at least one color channel of the sample image, determine the gain parameter of the target color channel as a first initial gain parameter; Performing numerical adjustment processing on the first initial gain parameter to obtain a first adjusted gain parameter of the target color channel; A ratio of the first initial gain parameter to the first adjusted gain parameter is determined as a sample offset coefficient of the target color channel.

3. The method according to claim 2, characterized in that The step of numerically adjusting the first initial gain parameter to obtain a first adjusted gain parameter of the target color channel includes: Generate a first value within a preset value range, and perform the value adjustment process on the first initial gain parameter according to the first value to obtain a first adjusted gain parameter of the target color channel; The numerical value adjustment process is randomly selected from the adjustment types, or the numerical value adjustment process is predetermined from the adjustment types, and the adjustment types include: increasing the numerical value, decreasing the numerical value, and keeping the numerical value unchanged.

4. The method according to claim 2 or 3, characterized in that: The step of performing an offset process on the tone parameter of the first processed image according to the sample offset coefficient to obtain a reference tone parameter includes: For any of the target color channels, among the gain parameters of at least one color channel of the first processed image, determining the gain parameter of the target color channel as a second initial gain parameter; Determining a second adjustment gain parameter of the target color channel according to a product of the sample offset coefficient of the target color channel and the second initial gain parameter; Among the gain parameters of at least one color channel of the first processed image, the gain parameter of the target color channel is replaced with the second adjustment gain parameter to obtain the reference hue parameter.

5. An image processing method, characterized in that: include: Acquire an original image captured by a camera, and acquire a hue shift coefficient of at least one target color channel, wherein the hue shift coefficient is used to indicate that a hue parameter corresponding to the target color channel is to be shifted; The original image and the hue shift coefficient of the at least one target color channel are input into an image processing model to obtain a target image output by the image processing model, wherein the image processing model is trained according to the method according to any one of claims 1-4.

6. The method according to claim 5, characterized in that The step of obtaining a hue shift coefficient of at least one target color channel includes: Inputting a test input image into an image processing model to obtain a test output image output by the image processing model; A hue shift coefficient of the at least one target color channel is determined according to the hue parameter of the test input image and the hue parameter of the test output image.

7. The method according to claim 5 or 6, characterized in that: The processing unit in the image processing model includes at least one of the following: A black level correction unit, an automatic white balance unit, an integrated processing unit, a local color correction unit, and a lens shading correction unit, wherein the integrated processing unit is used to implement at least one of the following functions: noise reduction processing, demosaicing processing, and high dynamic range processing; The local color correction unit is used to perform color cast compensation on the intermediate image output by the integrated processing unit according to the compensation matrix.

8. The method according to claim 7, characterized in that The method further comprises: During the processing of the image processing model, obtaining an intermediate image output by the integrated processing unit; Dividing the original image and the intermediate image into n image regions, wherein the n image regions in the original image correspond to the n image regions in the intermediate image in a one-to-one manner, and n is an integer greater than or equal to 1; For any of the image regions, determining a regional compensation value of at least one target color channel corresponding to the image region according to first pixel value information of the image region in the original image and second pixel value information of the image region in the intermediate image; A first compensation matrix of the at least one target color channel is determined according to the area compensation value of the at least one target color channel corresponding to each of the image areas.

9. The method according to claim 8, characterized in that The determining, according to the area compensation value of at least one target color channel corresponding to each of the image areas, a first compensation matrix of the at least one target color channel comprises: For any of the target color channels, the area compensation value of each of the image areas is set to the pixel compensation value corresponding to each pixel point in each of the image areas; The first compensation matrix is ​​determined by a matrix composed of pixel compensation values ​​corresponding to each pixel point in the intermediate image.

10. The method according to claim 9, characterized in that After determining the first compensation matrix by forming a matrix composed of pixel compensation values ​​corresponding to each pixel point in each image area, the method further includes: For a transition region between any two adjacent image regions, smoothing the pixel compensation values ​​of the pixel points in the transition region to obtain smoothed pixel compensation values ​​of the pixel points in the transition region; The first compensation matrix is ​​updated according to the smoothed pixel compensation values ​​of the pixels in the transition area.

11. The method according to claim 8, characterized in that The pixel value information includes an average pixel value of a first color channel of each pixel point in the image area, an average pixel value of a second color channel of each pixel point in the image area, and an average pixel value of a third color channel of each pixel point in the image area; The determining, according to the first pixel value information of the image area in the original image and the second pixel value information of the image area in the intermediate image, a regional compensation value of at least one target color channel corresponding to the image area comprises: Determine a ratio of an average pixel value of the first color channel to an average pixel value of the second color channel in the first pixel value information as a first ratio; Determine a ratio of an average pixel value of the first color channel to an average pixel value of the second color channel in the second pixel value information as a second ratio; The ratio of the first ratio to the second ratio is determined as the regional compensation value of the target color channel.

12. The method according to any one of claims 8 to 11, characterized in that: The method further comprises: Acquire a preset second compensation matrix, wherein the matrix size of the second compensation matrix is ​​consistent with the image size of the intermediate image, and the compensation value of each matrix position in the second compensation matrix is ​​proportional to a first distance corresponding to the matrix position, and the first distance is the distance between the matrix position and the center position of the second compensation matrix; Multiplying the first compensation matrix and the second compensation matrix at corresponding positions to obtain a target compensation matrix; According to the target compensation matrix, compensation processing is performed on the pixel value of each pixel point of the intermediate image.

13. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method according to any one of claims 1 to 12.

14. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 12.

Images