A training method of a parameter prediction network, an image color correction method and device

Abstract

The embodiment of the application provides a kind of parameter prediction network training method, image color correction method and device, it is related to digital image processing, computational photography, low-illumination imaging and computer vision technical field, technical scheme includes: obtaining training sample and training label.Afterwards, the parameter correction graph corresponding to the training sample is predicted by parameter prediction network, and the parameter correction graph represents the local color correction matrix corresponding to the multiple image regions included in the training sample.Then, based on the parameter correction graph, the color of the training sample is corrected to obtain a corrected image.The network parameters of the parameter prediction network are adjusted using the image loss determined based on the corrected image and the training label, and the step of predicting the parameter correction graph corresponding to the training sample by the parameter prediction network is returned until the parameter prediction network converges.The difference between the corrected image and the color image consistent with the habit of human eye vision is reduced.

Description

Technical Field

[0001] This application relates to the fields of digital image processing, computational photography, low-light imaging and computer vision, and in particular to a training method for a parameter prediction network, an image color correction method and apparatus. Background Technology

[0002] In addition to responding to visible light, image sensors in conventional color cameras typically also respond to near-infrared light with wavelengths above approximately 700 nanometers (nm). This can cause the image to appear reddish, purplish, or exhibit other non-linear color casts. To obtain color images that conform to human visual perception, an infrared cutoff filter is usually placed in front of the camera lens or sensor, putting the camera in a near-infrared light-blocking state and thus suppressing near-infrared light from entering the sensor. However, the infrared cutoff filter also reduces the total amount of light entering the sensor, resulting in blurred dark areas, increased noise, and loss of image detail in images taken in low-light environments such as at night or outdoors.

[0003] To enable cameras to obtain color images with rich detail and conforming to human visual perception in low-light environments, infrared cut-off filters are typically not used. This allows the camera to operate in visible and near-infrared light transmission mode, thereby increasing the effective light transmission of the sensor in low-light conditions, improving the visibility of structure and texture in dark areas of the image, and performing color correction on the raw image obtained by the sensor to reduce non-linear color casts. Existing color correction schemes use a fixed Color Correction Matrix (CCM) to correct the color of the raw image. This scheme assumes that every pair of pixels at the same location in the raw image and the color image conforming to human visual perception follows a uniform color mapping relationship. However, in actual low-light scenes, the light source spectrum, the near-infrared reflectivity of object materials, spatial illuminance, and the mixing ratio of visible and near-infrared light all have significant spatial differences. Even objects with similar visible light colors may have significantly different near-infrared reflectivity characteristics, resulting in not every pair of pixels at the same location in the raw image and the color image conforming to human visual perception following a uniform color mapping relationship. Therefore, the corrected image obtained by this color correction scheme is prone to residual color cast or overcorrection, and there is a large gap between it and the color image that conforms to the visual habits of the human eye. Summary of the Invention

[0004] The purpose of this application is to provide a training method for a parameter prediction network, an image color correction method, and an apparatus to reduce the gap between the corrected image and a color image that conforms to human visual habits. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a method for training a parameter prediction network, the method comprising:

[0006] Acquire training samples and training labels. The training samples are images of the sample environment captured by the image acquisition device with short exposure under visible and near-infrared light conditions. The training labels are images of the sample environment captured by the image acquisition device with long exposure under near-infrared light cutoff conditions. The illuminance of the sample environment is lower than a preset illuminance.

[0007] The parameter correction map corresponding to the training sample is predicted by the parameter prediction network, and the parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the training sample.

[0008] Based on the parameter correction map, the training samples are color corrected to obtain the corrected image;

[0009] Based on the corrected image and the training labels, determine the image loss;

[0010] The network parameters of the parameter prediction network are adjusted using the image loss, and the step of predicting the parameter correction map corresponding to the training sample through the parameter prediction network is returned until the parameter prediction network converges, thus obtaining the trained parameter prediction network.

[0011] In some embodiments, predicting the parameter correction map corresponding to the training samples using a parameter prediction network includes:

[0012] The training samples are downsampled, and the downsampled images are input into the parameter prediction network.

[0013] The input data is feature extracted by the feature extraction module of the parameter prediction network, the input features are downsampled by the downsampling module of the parameter prediction network, and a parameter correction map of a specified dimension is output by the output layer of the parameter prediction network. The feature extraction module includes a convolutional layer and an activation layer. The height of the specified dimension is smaller than the height of the training sample, the width is smaller than the width of the training sample, and the number of channels is 9.

[0014] In some embodiments, the parameter correction map includes feature maps of 9 channels; the step of color correction of the training samples based on the parameter correction map to obtain a corrected image includes:

[0015] The resolution of the parameter correction map is converted to the resolution of the training sample; for each pixel in the training sample, a local color correction matrix is ​​constructed using the feature points of the corresponding 9 channels of the pixel to perform color correction on the pixel, resulting in a corrected image composed of the corrected pixels; wherein the position of the pixel in the training sample is the same as the position of the corresponding feature point in the feature map after resolution conversion; or...

[0016] For each pixel in the training sample, a local color correction matrix is ​​constructed using the feature points of the nine channels corresponding to the image region where the pixel is located. The pixel is then color-corrected to obtain a corrected image composed of the corrected pixels. The position of the image region in the training sample is the same as the position of the feature points corresponding to the image region in the feature map.

[0017] In some embodiments, determining the image loss based on the corrected image and the training labels includes:

[0018] The color reconstruction loss is determined based on the color difference between pixels at the same position in the corrected image and the training label.

[0019] The structure preservation loss is determined based on the pixel mean and pixel standard deviation of the corrected image and the training label, as well as the pixel covariance between the corrected image and the training label.

[0020] The image loss is determined based on the color reconstruction loss and the structure preservation loss.

[0021] In some embodiments, determining the image loss based on the color reconstruction loss and the structure preservation loss includes:

[0022] An additional loss is determined, which includes at least one of the following: a chromatic difference loss determined based on the chromatic difference between the corrected image and the training label; a spatial smoothing loss determined based on the difference between adjacent feature points in the intermediate feature map; a brightness preservation loss determined based on the brightness difference between the corrected image and the training sample; and a regularization loss determined based on the norm of the network layer weights of the parameter prediction network. The intermediate feature map is an intermediate result obtained by the parameter prediction network processing the training sample.

[0023] The weighted sum of the color reconstruction loss, the structure preservation loss, and the additional loss is taken as the image loss.

[0024] Secondly, embodiments of this application provide an image color correction method, the method comprising:

[0025] Acquire an image to be corrected, which is an image of the target environment captured by an image acquisition device under visible and near-infrared light conditions;

[0026] The parameter prediction network predicts the parameter correction map corresponding to the image to be corrected, wherein the parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the image to be corrected, and the parameter prediction network is trained by the method described in any one of the first aspects.

[0027] Based on the parameter correction map, color correction is performed on the image to be corrected to obtain a corrected image.

[0028] Thirdly, embodiments of this application provide a training apparatus for a parameter prediction network, the apparatus comprising:

[0029] The acquisition module is used to acquire training samples and training labels. The training samples are images of the sample environment captured by the image acquisition device under visible and near-infrared light conditions with short exposure. The training labels are images of the sample environment captured by the image acquisition device under near-infrared light cutoff conditions with long exposure. The illuminance of the sample environment is lower than the preset illuminance.

[0030] The prediction module is used to predict the parameter correction map corresponding to the training sample through the parameter prediction network. The parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the training sample.

[0031] The correction module is used to perform color correction on the training samples based on the parameter correction map to obtain a corrected image;

[0032] A determination module is used to determine the image loss based on the corrected image and the training labels;

[0033] The adjustment module is used to adjust the network parameters of the parameter prediction network using the image loss, and to call the prediction module to perform the step of predicting the parameter correction map corresponding to the training sample through the parameter prediction network until the parameter prediction network converges, thus obtaining the trained parameter prediction network.

[0034] In some embodiments, the prediction module is specifically used for:

[0035] The training samples are downsampled, and the downsampled images are input into the parameter prediction network.

[0036] The input data is feature extracted by the feature extraction module of the parameter prediction network, the input features are downsampled by the downsampling module of the parameter prediction network, and a parameter correction map of a specified dimension is output by the output layer of the parameter prediction network. The feature extraction module includes a convolutional layer and an activation layer. The height of the specified dimension is smaller than the height of the training sample, the width is smaller than the width of the training sample, and the number of channels is 9.

[0037] In some embodiments, the parameter correction map includes feature maps of 9 channels; the correction module is specifically used for:

[0038] The resolution of the parameter correction map is converted to the resolution of the training sample; for each pixel in the training sample, a local color correction matrix is ​​constructed using the feature points of the corresponding 9 channels of the pixel to perform color correction on the pixel, resulting in a corrected image composed of the corrected pixels; wherein the position of the pixel in the training sample is the same as the position of the corresponding feature point in the feature map after resolution conversion; or...

[0039] For each pixel in the training sample, a local color correction matrix is ​​constructed using the feature points of the nine channels corresponding to the image region where the pixel is located. The pixel is then color-corrected to obtain a corrected image composed of the corrected pixels. The position of the image region in the training sample is the same as the position of the feature points corresponding to the image region in the feature map.

[0040] In some embodiments, the determining module is specifically used for:

[0041] The color reconstruction loss is determined based on the color difference between pixels at the same position in the corrected image and the training label.

[0042] The structure preservation loss is determined based on the pixel mean and pixel standard deviation of the corrected image and the training label, as well as the pixel covariance between the corrected image and the training label.

[0043] The image loss is determined based on the color reconstruction loss and the structure preservation loss.

[0044] In some embodiments, the determining module is specifically used for:

[0045] An additional loss is determined, which includes at least one of the following: a chromatic difference loss determined based on the chromatic difference between the corrected image and the training label; a spatial smoothing loss determined based on the difference between adjacent feature points in the intermediate feature map; a brightness preservation loss determined based on the brightness difference between the corrected image and the training sample; and a regularization loss determined based on the norm of the network layer weights of the parameter prediction network. The intermediate feature map is an intermediate result obtained by the parameter prediction network processing the training sample.

[0046] The weighted sum of the color reconstruction loss, the structure preservation loss, and the additional loss is taken as the image loss.

[0047] Fourthly, embodiments of this application provide an image color correction device, the device comprising:

[0048] The acquisition module is used to acquire the image to be corrected, which is an image of the target environment captured by the image acquisition device under visible light and near-infrared light conditions with a first illuminance, where the first illuminance is less than a preset illuminance.

[0049] The prediction module is used to predict the parameter correction map corresponding to the image to be corrected through a parameter prediction network. The parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the image to be corrected. The parameter prediction network is trained by the method described in any one of the first aspects.

[0050] The correction module is used to perform color correction on each image region included in the image to be corrected based on the parameter correction map, so as to obtain a corrected image.

[0051] Fifthly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0052] Memory, used to store computer programs;

[0053] A processor, when executing a program stored in memory, implements the method described in either the first or second aspect.

[0054] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in either the first or second aspect.

[0055] In a seventh aspect, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in either the first or second aspect described above.

[0056] Beneficial effects of the embodiments in this application:

[0057] The parameter prediction network training method, image color correction method, and apparatus provided in this application, based on the parameter correction map corresponding to the training samples predicted by the parameter prediction network, perform color correction on the training samples to obtain a corrected image, and adjust the network parameters of the parameter prediction network based on the corrected image and the image loss determined by the training label. The training samples are images of the sample environment captured by an image acquisition device under short exposure in visible and near-infrared light conditions, while the training labels are images of the sample environment captured by the image acquisition device under long exposure in near-infrared light cutoff conditions. The illumination of the sample environment is lower than a preset illumination level; that is, the training samples are color-biased images with visible and near-infrared light responses captured under low-light conditions, while the training labels are color images conforming to human visual habits. The parameter correction map predicted by the parameter prediction network trained in this way can correct the input image into a color image closer to human visual habits. Simultaneously, since the parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the training samples, each image region within the image can be color-corrected according to a local color correction matrix more suitable for that image region, improving the flexibility of color correction and thus reducing the gap between the corrected image and the color image conforming to human visual habits.

[0058] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0060] Figure 1 A flowchart illustrating a training method for a parameter prediction network provided in an embodiment of this application;

[0061] Figure 2 The response curves of each color channel of a sensor to light of different wavelengths are provided for embodiments of this application.

[0062] Figure 3 A comparison image of the same scene captured by a camera under different conditions, provided as an embodiment of this application;

[0063] Figure 4 This is an exemplary schematic diagram illustrating the image processing procedure of a parameter prediction network provided in an embodiment of this application.

[0064] Figure 5 A comparison image before and after color correction is provided for an embodiment of this application;

[0065] Figure 6 A flowchart of an image color correction method provided in an embodiment of this application;

[0066] Figure 7 A schematic diagram of the structure of a training device for a parameter prediction network provided in an embodiment of this application;

[0067] Figure 8 This is a schematic diagram of the structure of an image color correction device provided in an embodiment of this application;

[0068] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0070] This application provides a method for training a parameter prediction network, applicable to electronic devices, such as smart cameras, mobile phones, tablets, or wearable devices with image acquisition and processing capabilities. Figure 1 As shown, the training method for the parameter prediction network provided in this application embodiment includes S101 to S105.

[0071] S101. Obtain training samples and training labels. The training samples are images of the sample environment captured by the image acquisition device under visible and near-infrared light conditions using short exposure. The training labels are images of the sample environment captured by the image acquisition device under near-infrared light cutoff conditions using long exposure, where the illuminance of the sample environment is lower than a preset illuminance.

[0072] S102. Predict the parameter correction map corresponding to the training sample using a parameter prediction network. The parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the training sample.

[0073] S103. Based on the parameter correction map, perform color correction on the training samples to obtain the corrected image.

[0074] S104. Determine the image loss based on the corrected image and the training labels.

[0075] S105. Adjust the network parameters of the parameter prediction network using image loss, and return to S102 until the parameter prediction network converges, thus obtaining the trained parameter prediction network.

[0076] The parameter prediction network training method provided in this application provides a method for color correction of training samples based on the parameter correction map corresponding to the training samples predicted by the parameter prediction network, resulting in a corrected image. The network parameters of the parameter prediction network are then adjusted based on the corrected image and the image loss determined by the training label. The training samples are images of the sample environment captured by an image acquisition device under short exposure in visible and near-infrared light conditions. The training label is an image of the sample environment captured by the image acquisition device under long exposure in near-infrared light cutoff conditions, and the illumination of the sample environment is lower than a preset illumination level. In other words, the training samples are color-biased images with visible and near-infrared light responses captured under low-light conditions, while the training label is a color image that conforms to human visual habits. The parameter correction map predicted by the parameter prediction network trained in this way can correct the input image into a color image that is closer to human visual habits. Furthermore, since the parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the training sample, each image region within the image can be color-corrected according to a local color correction matrix that better suits that image region, improving the flexibility of color correction and thus reducing the gap between the corrected image and the color image that conforms to human visual habits.

[0077] The following are Figure 1 The implementation methods of each step are explained in detail.

[0078] For S101, when the camera has no infrared cut-off filter set or removed, or when the infrared cut-off filter is set to bypass, allowing the camera to pass through both visible and near-infrared light, a short-exposure method is used to capture the sample environment, resulting in a color-cast image that includes both visible and near-infrared light responses. This color-cast image is then used as a training sample. The illuminance of the sample environment is lower than a preset illuminance, meaning the sample environment is a low-illuminance environment.

[0079] The training samples are captured using short exposures in low-light scenes to more closely resemble color-shifted images captured in real-world scenarios. For example, the exposure time for a short exposure is 30 milliseconds (ms). Furthermore, because the training samples are captured with the camera in visible and near-infrared light conditions, the light intake is higher, resulting in richer image details in the darker areas.

[0080] See Figure 2 , Figure 2The horizontal axis represents the wavelength of light, in nm; the vertical axis represents the relative response of the image sensor, that is, the ratio of the electrical signal response value output by the image sensor to the ideal linear response value, in arbitrary units (au). Figure 2 The solid line curve represents the response curve of the image sensor's blue channel to different wavelengths of light; the curve composed of dots represents the response curve of the image sensor's red channel to different wavelengths of light; and the curve composed of short line segments represents the response curve of the image sensor's green channel to different wavelengths of light. Figure 2 It can be seen that the three color channels of the image sensor have similar response values ​​to near-infrared light with a longer wavelength, making it difficult to achieve accurate color differentiation.

[0081] Therefore, in order to obtain more accurate color training labels, this embodiment of the application equips the camera with an infrared cutoff filter, putting the camera in a near-infrared light cutoff state, that is, in a normal color imaging state where the near-infrared light band above approximately 700nm is cut off. At this time, the visible light is allowed, and a long exposure method is used to photograph the sample environment, obtaining a normal color image that includes visible light response and has no obvious near-infrared light response. This image is used as the training label for the training sample. The training label can also be called the supervised ground truth.

[0082] The training labels are captured using long exposure, which ensures sufficient light intake and allows for richer detail in the labels. For example, the long exposure duration is 1 second (s).

[0083] Optionally, to further reduce the noise contained in the training labels, multiple images of the sample environment can be continuously captured using long exposure when the camera is in near-infrared light cutoff mode. After the multiple images are registered, they can be superimposed or fused to reduce the noise in the images, improve the signal-to-noise ratio, and make the colors of the processed images closer to the colors that conform to human visual habits under normal illumination.

[0084] Images captured in low-light environments often have significant noise and inaccurate colors. However, this embodiment of the application uses a near-infrared light cutoff and long exposure method to capture training labels in low-light environments. This increases the light transmittance, reduces noise in the image, and minimizes color cast caused by near-infrared light, thus ensuring the image quality of the training labels. This method is then used to train the parameter prediction network, which is beneficial for the network to learn a more stable color mapping relationship.

[0085] For example, see Figure 3 , Figure 3The left image shows the sample environment captured by a camera using a long exposure under near-infrared light cutoff conditions. The middle image shows the sample environment captured by a camera using a short exposure under near-infrared light cutoff conditions. The right image shows the sample environment captured by a camera using a short exposure under visible and near-infrared light conditions. Comparing the left and middle images, it can be seen that the image captured by the camera using the long exposure method has less noise and higher color reproduction accuracy. Comparing the middle and right images, it can be seen that although there is some color distortion when shooting under visible and near-infrared light conditions, the greater light transmission results in more detail in the shadows. Therefore, color correction is performed on the right image, resulting in an image that contains richer details and has higher color accuracy. Therefore, using the left image as the training label and the right image as the training sample allows the parameter prediction network to predict a more suitable parameter correction image for the training sample, making the color-corrected training sample closer to the training label.

[0086] To ensure consistency between the sample scenes used when shooting training samples and training labels, the camera and sample scene are kept still during image acquisition. Global or local registration, cropping, and size unification are performed on the training samples and training labels to reduce errors in the shooting angle of the training samples and training labels caused by slight lens movement due to switching infrared cutoff filters or environmental interference.

[0087] It should be noted that both the training samples and training labels are images in the Linear RGB domain. The raw images obtained by the camera's sensor can first undergo at least one ISP process, such as black level correction, bad pixel correction, demosaicing, white balance, or gain normalization, to obtain an image in the Linear RGB domain. If the image after ISP processing is not in the Linear RGB domain, for example, if the gamma encoding or tone mapping included in the ISP process converts the original image into a non-linear RGB image, then a linear transformation such as inverse gamma transform or inverse tone mapping is performed on the ISP-processed image to restore or approximately restore the image to the Linear RGB domain.

[0088] Since the pixel values ​​of the image output by the sensor are directly proportional to the intensity of the light radiation incident on the sensor, setting the training samples and training labels as Linear RGB images can give the images used for training clear physical meaning and enable the parameter prediction network to learn parameter correction maps that conform to the sensor imaging rules, thereby reducing the contradiction between the mapping relationship between the training samples and the correction images represented by the parameter correction maps and the physical imaging logic.

[0089] After obtaining the training samples and training labels in the Linear RGB domain, in order to eliminate the influence of the brightness difference between the training samples and the training labels on the parameter prediction network's learning of the parameter correction map, this embodiment of the application further normalizes the pixel color vector of the training samples before inputting them into the parameter prediction network, and normalizes the pixel color vector of the training labels before calculating the loss based on the training labels. The normalization process removes the amplitude / brightness information of the RGB pixels in the image, retaining only the relative direction between the three RGB channels, which mainly retains the color ratio information.

[0090]

[0091]

[0092] in, This represents the color vector of a pixel in the training sample / training label. This represents the red channel value of a pixel in the training sample / training label. This represents the green channel value of a pixel in the training sample / training label. This represents the blue channel value of a pixel in the training sample / training label. This represents the normalized color vector of pixels in the training sample / training label.

[0093] For any pixel, its three color channel values ​​constitute a color vector, and the norm of the color vector is used to normalize the color vector of each color channel value at the pixel level. This reduces the impact of different exposure times, exposure gains, and local brightness differences on the learning of color mapping parameters in the parameter correction map, so that the parameter prediction network mainly learns the channel ratio shift introduced by near-infrared light.

[0094] The parameter correction map output by the parameter prediction network includes a local CCM color correction matrix. The result of color correction on the normalized input data is color proportion information. The result mainly includes the corrected color channel relationships, but its brightness amplitude does not correspond to the actual brightness of the original image to be corrected. That is, it differs from the actual brightness of the training samples before normalization. Therefore, brightness recovery is still needed based on the brightness information of the training samples before normalization. The final result is obtained after adding brightness information. .

[0095]

[0096] in, This represents the image brightness based on the training samples before normalization. This represents the image brightness after color correction, and the image brightness... Determined based on preset weights for multiple color channels, for example .

[0097] The method for correcting the prediction parameters for S102 includes steps 1 and 2.

[0098] Step 1: Downsample the training samples and input the downsampled images into the parameter prediction network.

[0099] Step 2: The feature extraction module of the parameter prediction network extracts features from the input data, the downsampling module of the parameter prediction network downsamples the input features, and the output layer of the parameter prediction network outputs a parameter correction map of the specified dimension. The feature extraction module includes convolutional layers and activation layers. The height and width of the specified dimension representation are smaller than the height and width of the training samples, and the number of channels is 9.

[0100] Compared to end-to-end image generation networks, which require an input image to be corrected and output a corrected image, resulting in a complex network structure and high computational cost, making them difficult to deploy on low-computing-power devices, the parameter prediction network in this embodiment processes downsampled training samples. Furthermore, the network can perform further downsampling during processing, reducing the amount of data processed and thus the computational cost. Moreover, the feature extraction layer of the parameter prediction network in this embodiment includes convolutional and activation layers, resulting in a simpler network structure. The parameter prediction network in this embodiment has a low computational cost; for example, processing a 1080×1920 resolution image requires only 4.6 billion Giga Floating-point Operations (GFLOPs). Therefore, it can be deployed on devices with lower computing power to achieve real-time or near-real-time image correction, broadening the application scope of this embodiment.

[0101] Furthermore, end-to-end image generation networks suffer from poor output interpretability, alteration of image brightness or structural details, generation of pseudo-textures, and image instability after correcting consecutive images. In contrast, the parameter prediction network in this embodiment does not directly output the corrected image (i.e., it does not regenerate image structure and details). Instead, it outputs a parameter correction map representing the local color correction matrices corresponding to multiple image regions included in the training samples. This improves the interpretability of the output results. Using local color correction matrices to correct the image reduces the impact on image brightness, geometric structure, and texture details, lowers the probability of pseudo-textures, and reduces image instability issues such as inter-frame texture flicker caused by correcting consecutive images.

[0102] In this embodiment, training samples at their original resolution can be input into the parameter prediction network; alternatively, the training samples can be downsampled, where the resolution of the downsampled image is lower than the original resolution of the training samples, and the downsampled image is input into the parameter prediction network, thereby reducing the amount of data processed by the parameter prediction network. Furthermore, before inputting training samples into the parameter prediction network or before downsampling the training samples, the training samples in the Linear RGB domain are normalized.

[0103] The parameter prediction network comprises multiple feature extraction modules, at least one downsampling module, and an output layer. The first feature extraction module extracts features from the input image, while the remaining modules extract features from the input. The downsampling module downsamples the output features of one feature extraction module and inputs the downsampled features into another feature extraction module or the output layer. The feature extraction modules consist of multiple cascaded network layer combinations, each including a convolutional layer and an activation layer. The downsampling module includes a convolutional layer, allowing the overall parameter prediction network to be constructed by stacking convolutional and activation layers.

[0104] In one possible implementation, the parameter prediction network may further include a resolution conversion module following the output layer. This module converts the resolution of the parameter correction map output by the output layer to the original resolution of the training samples. For example, the resolution conversion module converts the height of the parameter correction map to the height of the training samples and the width of the parameter correction map to the width of the training samples through upsampling or block mapping.

[0105] For example, the parameter prediction network includes a first feature extraction module, a first downsampling module, a second feature extraction module, a second downsampling module, an output layer, and a resolution conversion module.

[0106] Combination Figure 4 The training samples have a resolution of H×W. The first feature extraction module is used to extract shallow features from the input training samples, such as edge, texture and color features, and output features with a resolution of H×W.

[0107] The first downsampling module is used to downsample the features with a resolution of H×W output by the first feature extraction module to obtain features with a resolution of H / 2×W / 2.

[0108] The second feature extraction module is used to extract deeper features from the features output by the first downsampling module, such as semantic features like scene structure, and output features with a resolution of H / 2×W / 2.

[0109] The second downsampling module is used to downsample the features with a resolution of H / 2×W / 2 output by the second feature extraction module to obtain features with a resolution of H / 4×W / 4.

[0110] The output layer is used to output a parameter correction map with a resolution of H / 4×W / 4 and 9 channels.

[0111] The resolution conversion module is used to convert the resolution of the parametric correction map to H×W.

[0112] In this embodiment of the application, the kernel size, number of channels, number of layers, stride, and activation function of the activation layer in the parameter prediction network can all be flexibly set according to the actual application scenario.

[0113] In this embodiment, the parameter correction map includes feature maps of nine channels. Features at the same position in these nine feature maps form a 3×3 local color correction matrix. Each diagonal element of the local color correction matrix represents the gain coefficient for one of the RGB color channels, used to control the saturation of that color channel. Each off-diagonal element of the local color correction matrix represents the influence of one of the RGB color channels on another, used to control the degree of color cast in each color channel.

[0114] Based on this, the above-mentioned S103 performs color correction on the training samples based on the parameter correction map, and the corrected image can be obtained in two ways: Method 1 and Method 2.

[0115] Method 1 for implementing S103: Convert the resolution of the parameter correction map to the resolution of the training sample; for each pixel in the training sample, use the local color correction matrix constructed from the feature points of the 9 channels corresponding to the pixel to perform color correction on the pixel, and obtain the corrected image composed of each pixel; wherein, the position of the pixel in the training sample is the same as the position of the feature point corresponding to the pixel in the feature map after resolution conversion.

[0116] The resolution of the parametrically corrected image can be converted to the original resolution of the training samples using a resolution conversion module included in the parametric correction network. Alternatively, when the parametric correction network does not include a resolution conversion module, the parametrically corrected image can be upsampled. For example, upsampling can be performed using nearest neighbor interpolation, bilinear interpolation, or other spatial interpolation methods. Frame interpolation makes the transition between adjacent feature points in the resolution-converted parametrically corrected image smoother, which in turn makes the transition between adjacent local color correction matrices smoother, thereby reducing the occurrence of color abrupt changes at the boundaries of the corresponding image patches in the training samples for each feature point in the original parametrically corrected image.

[0117] For a pixel at position (x, y) in the training samples, the color vector of that pixel is:

[0118]

[0119] in, Let be the color vector of the pixel at position (x, y). Let be the red channel color component of the pixel at position (x, y). Let be the green channel color component of the pixel at position (x, y). Let T represent the blue channel color component of the pixel at position (x,y), where T denotes transpose.

[0120] Using the local color correction matrix After color correction, the following was obtained:

[0121]

[0122] in, This is the color vector of the pixel at position (x, y) after color correction. Let be the local color correction matrix corresponding to the pixel at position (x, y). .

[0123] Method 2 for implementing S103: For each pixel in the training sample, a local color correction matrix is ​​constructed using the feature points of the 9 channels corresponding to the image region where the pixel is located. This matrix is ​​then used to correct the color of the pixel, resulting in a corrected image composed of all the corrected pixels. The position of the image region in the training sample is the same as the position of the corresponding feature point in the feature map.

[0124] The training samples include the same number of image regions as each feature map in the parameter correction map, and each image region has the same size.

[0125] For example, if the resolution of the parameter correction map is H / 4×W / 4 and the resolution of the training samples is H×W, then the training samples are divided into image regions with a resolution of 4×4. There is no overlap between each image region, and the union of all image regions is the training sample.

[0126] This application embodiment can downsample the training samples before inputting them into the parameter prediction model for predicting the parameter correction map. However, color correction is performed on the training samples at the original resolution using the parameter correction map, thus balancing the global and local receptive fields and computational efficiency of the parameter prediction network. Simultaneously, color correction preserves the original image details of the training samples, reducing detail loss. This application embodiment can convert the resolution of the parameter correction map to the resolution of the training samples, thereby performing targeted color correction on each pixel of the training samples, improving the accuracy of color correction for the training samples. Alternatively, this application embodiment can utilize the local color correction matrix corresponding to the image region where the pixel in the training sample is located for color correction, achieving region-based color correction of the training samples, balancing the flexibility and efficiency of color correction, i.e., improving both accuracy and efficiency.

[0127] For S104, the method for determining image loss based on the corrected image and training labels includes steps one through three.

[0128] Step 1: Determine the color reconstruction loss based on the color difference between pixels at the same location in the corrected image and the training label.

[0129] Step 2: Determine the structure preservation loss based on the pixel mean and standard deviation of the corrected image and the training label, as well as the pixel covariance between the corrected image and the training label.

[0130] Step 3: Determine the image loss based on the color reconstruction loss and structure preservation loss.

[0131] This application embodiment can determine image loss based on color reconstruction loss and structure preservation loss, and use the image loss to adjust the network parameters of the parameter prediction network. The color reconstruction loss can characterize the color difference between pixels at the same position in the corrected image and the training label, and the structure preservation loss can characterize the structural similarity between the corrected image and the training label. This achieves the goal of reducing the color difference between the corrected image and the training label and improving the structural similarity by training the parameter prediction network. Through training, the parameter prediction network can predict a more accurate parameter correction map for the input image, so that the input image can be color corrected more accurately based on the parameter correction map. That is, the difference between the corrected image and the color image that conforms to the human visual habit is smaller.

[0132] The color reconstruction loss is:

[0133]

[0134] in, For color reconstruction loss, To correct the total number of pixels in the image, To train the RGB color vector of the i-th pixel in the label, To correct the RGB color vector of the i-th pixel in the image, This indicates taking the absolute value.

[0135] The structural retention loss is:

[0136]

[0137]

[0138]

[0139]

[0140] in, To preserve structural losses, To correct the image similarity between the image and the training labels, To correct the brightness similarity between the image and the training labels, To correct for contrast and structural similarity between the image and the training labels, To train the pixel mean of the labels, To correct the pixel mean of an image, which reflects the image's brightness, To determine the pixel standard deviation of the training labels, To correct the pixel standard deviation of an image, which reflects image contrast. To correct for the pixel covariance between the image and the training label, which reflects the structural correlation between images, and It is a zero-prevention constant, used to prevent the denominator in the calculation formula from approaching 0. , L represents the upper limit of the dynamic range of pixel values, that is, the maximum value of a single color channel, for example, L=255. and For example, a preset constant. , . The closer to 1, The smaller the value, the higher the structural similarity between the corrected image and the training label.

[0141] After determining the color reconstruction loss and structure preservation loss, the two can be directly summed or weighted summed as the image loss.

[0142] Alternatively, an additional loss can be determined, which is a weighted sum of the color reconstruction loss, structure preservation loss, and additional loss, as the image loss.

[0143] The additional loss includes at least one of the following: chromatic difference loss determined based on the chromatic difference between the corrected image and the training label; spatial smoothing loss determined based on the difference between adjacent feature points in the intermediate feature map; brightness preservation loss determined based on the brightness difference between the corrected image and the training sample; and regularization loss determined based on the norm of the network layer weights of the parameter prediction network. The intermediate feature map is the intermediate result obtained by the parameter prediction network from processing the training samples.

[0144] By combining color reconstruction loss, structure preservation loss, and additional loss, the parameter prediction network can not only reduce the color difference between the corrected image and the training label and improve their structural similarity, but also reduce the chromaticity difference between the corrected image and the training label, reduce the brightness difference between the corrected image and the training samples, and ensure the smoothness of the intermediate feature map. This prevents the network layer weights of the parameter prediction network from overfitting, further improving the accuracy of the parameter correction map predicted by the parameter prediction network for the input image, and enabling more accurate color correction of the input image based on the parameter correction map.

[0145] The chromatic aberration loss between the corrected image and the training labels is:

[0146]

[0147] To compensate for color difference loss, To correct the total number of pixels in the image, This indicates that the image was extracted after being converted to the Lab color space. Channels and The chromaticity components of the channels, or the chromaticity components of the Cb and Cr channels extracted after converting the image to the YCbCr color space. Let i be the RGB color vector of the i-th pixel of the training label. To correct the RGB color vector of the i-th pixel in the image, This represents taking the square of the 2-norm. The Lab color space includes three channels, of which... Represents brightness. Representing the opposite hues of red and green, Representing the yellow-blue contrasting chromaticity, the YCbCr color space includes three channels: Y represents the luminance component, Cb represents the blue chromaticity component, and Cr represents the red chromaticity component.

[0148] Using chromatic aberration loss to train a parameter prediction network can reduce the chromaticity difference between the corrected image and the training label.

[0149] Intermediate feature maps can be the output of any feature extraction module in the parameter prediction network. For example, an intermediate feature map could be the output of the first feature extraction module in the parameter prediction network.

[0150] The spatial smoothing loss of intermediate feature maps can be expressed as a Total Variation (TV) loss:

[0151]

[0152] in, For spatial smoothing loss, The position in the intermediate feature map is Feature points, The position in the intermediate feature map is Feature points, This indicates taking the absolute value.

[0153] By using the spatial smoothing loss of intermediate feature maps to train the parameter prediction network, the smoothness of the intermediate feature maps generated by the parameter network can be constrained, reducing the artifacts caused by local drastic fluctuations in the intermediate feature maps on the corrected image.

[0154] The brightness preservation loss between the corrected image and the training samples is:

[0155]

[0156] in, To preserve brightness loss, To correct the total number of pixels in the image, This indicates that the image was extracted after being converted to the Lab color space. The channel component, or the Y channel component extracted after converting the image to the YCbCr color space. Both the Y channel component and the A channel component are luminance channel components. For the i-th pixel of the training sample, To correct the i-th pixel of the image, This indicates taking the 1 norm.

[0157] By using brightness preservation loss to train the parameter prediction network, the brightness difference between the training samples input to the parameter prediction network and the corrected image obtained by using the parameter correction map predicted by the parameter prediction network can be reduced, ensuring the brightness consistency between the training samples and the corrected image, and making the corrected image closer to the image captured in the real environment.

[0158] The regularization loss is expressed in the form of the L2 norm:

[0159]

[0160] in, For regularization loss, This refers to the set of weights for each layer of the parameter prediction network. Predict the k-th weight of the l-th layer of the network for the parameters.

[0161] By using regularized loss to train the parameter prediction network, we can constrain the excessive weight shift of the parameter prediction network and prevent overfitting.

[0162] For example, the image loss obtained by fusing various losses is:

[0163]

[0164] in, For image loss, The color reconstruction loss is weighted. For color reconstruction loss, To maintain the weight of the loss in the structure, To preserve structural losses, As the weight for color difference loss, To compensate for color difference loss, The weights for spatial smoothing loss, For spatial smoothing loss, Weights for maintaining brightness loss. To preserve brightness loss, The weights for the regularization loss are... This is the loss due to regularization. to It can be pre-configured manually based on the actual application scenario.

[0165] For S105, based on image loss, gradient descent and backpropagation are used to update the network parameters of the parameter prediction network, including the weights and biases of the network layers. Then, it is determined whether the convergence condition of the parameter prediction network is met. If the convergence condition is met, the parameter prediction network training is considered complete; otherwise, it returns to S102 and enters the next iteration, using the next batch of training samples and labels to continue training the parameter prediction network. The convergence condition includes at least one of the following: the number of iterations reaches a preset number; the image loss calculated in the current iteration is less than a preset loss; and the error between the image losses calculated in the most recent N iterations is less than a preset error.

[0166] The following uses a nighttime outdoor scene as an example to illustrate the complete process of the training method for the parameter prediction network provided in this application embodiment.

[0167] At 9 PM, the ambient illuminance of a park is approximately 0.1 lux, with even lower illuminance in vegetated areas. A single frame image of the park was captured with a camera under visible and near-infrared light conditions, using an exposure time of 30ms, and used as a training sample. Multiple frames were then captured continuously with the camera under near-infrared light cutoff conditions (above 700nm), using an exposure time of 1s. These multiple frames were registered and then superimposed to serve as training labels. Alternatively, a single frame image of the park could be captured with the camera under near-infrared light cutoff conditions (above 700nm), using an exposure time of 30ms, as a control image. This control image was compared with the color-corrected image to evaluate the color correction effect.

[0168] After the training samples are processed by ISP to form images in the Linear RGB domain, the images are normalized and then input into the parameter prediction network. During the processing of the input images, the parameter prediction network reduces the image resolution from H×W to H / 2×W / 2, and then to H / 4×W / 4. Finally, it outputs a parameter tensor with a resolution of H / 4×W / 4 and 9 channels, which is the parameter correction map.

[0169] Interpolate the parametric correction map to adjust its resolution to H×W, obtaining a pixel-by-pixel parametric correction map. For each pixel in the training samples, use a local color correction matrix constructed from the feature points of the corresponding 9 channels to perform color correction on that pixel, resulting in a corrected image. Alternatively, divide the training samples into 16 image regions of equal size. For each pixel in the training samples, use a local color correction matrix constructed from the feature points of the corresponding 9 channels of the image region to perform color correction on that pixel, resulting in a corrected image. For example, see... Figure 5 The left image is the training sample, and the right image is the corrected image after color correction.

[0170] During model training, the color reconstruction loss and structure preservation loss between the corrected image and the training labels are determined. Additionally, the chromatic aberration loss, spatial smoothing loss of intermediate feature maps, brightness preservation loss between the corrected image and the training samples, and regularization loss of the network layer weights in the parameter prediction network are also determined. The weighted sum of these losses—color reconstruction, structure preservation, chromatic aberration, spatial smoothing, brightness preservation, and regularization—is used as the image loss, and this image loss is then used to adjust the network parameters of the parameter prediction network.

[0171] Experimental observations revealed that removing the infrared cutoff filter, allowing the camera to capture images in visible and near-infrared light conditions, improves shadow detail in the captured images. However, it also introduces noticeable color cast. Using the parameter correction map obtained from the parameter prediction network provided in this application for color correction allows for targeted color correction of different regions of the image. Compared to applying a uniform global parameter correction matrix to all pixels, this application improves the flexibility of color correction. Furthermore, this application corrects the tinted portion rather than enhancing brightness, preserving the original structure and details of the image and reducing inter-frame flickering issues that occur after color correction of multiple consecutive frames.

[0172] Based on the same inventive concept, this application also provides an image color correction method applied to electronic devices, such as smart cameras, mobile phones, tablets, or wearable devices with image acquisition and processing capabilities. The parameter prediction network training method and the image color correction method can be applied to the same electronic device or to different electronic devices. Figure 6 As shown in the embodiments of this application, the image color correction method also includes steps S601 to S603.

[0173] S601. Acquire the image to be corrected. The image to be corrected is an image of the target environment captured by the image acquisition device under visible and near-infrared light conditions.

[0174] S602. Predict the parameter correction map corresponding to the image to be corrected using a parameter prediction network. The parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the image to be corrected. The parameter prediction network predicts the parameter correction map through... Figure 1 The parameter prediction network shown was trained using the training method described.

[0175] S603. Based on the parameter correction map, perform color correction on the image to be corrected to obtain the corrected image.

[0176] The image color correction method provided in this application uses a parameter prediction network to predict a parameter correction map corresponding to the image to be corrected, and then performs color correction on each image region included in the image to be corrected. Since the parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the corrected image, each image region in the image to be corrected can be color corrected according to a local color correction matrix that is more suitable for that image region, thus reducing the gap between the corrected image and a color image that conforms to human visual habits.

[0177] In this embodiment, the process of obtaining the image to be corrected is the same as the method of obtaining the training samples; the method of predicting the parameter correction map corresponding to the image to be corrected through the parameter prediction network is the same as the method of predicting the parameter correction map corresponding to the training samples through the parameter prediction network; and the method of color correction of the image to be corrected based on the parameter correction map is the same as the method of color correction of the training samples based on the parameter correction map. Figure 6 The specific implementation methods for each step can be found in the above descriptions, and will not be repeated here.

[0178] The target environment can be a low-light environment. Shooting in a low-light environment based on visible light and near-infrared light transmission results in higher light transmission and can retain the dark structure and texture information that is easily lost when near-infrared light is cut off. This makes the details of the obtained image to be corrected richer, and the image after color correction is closer to the image taken under normal lighting conditions, which has both rich details and accurate colors.

[0179] The target environment can also be a normal illumination environment. When the camera is in the state of visible light and near-infrared light, the image obtained in the normal illumination environment will also have color cast problem. The color can be corrected by the image color correction method provided in the embodiments of this application, so as to obtain a more accurate color image.

[0180] The image to be corrected acquired by S601 can be an image taken by the camera or a video frame captured by the camera.

[0181] After obtaining the corrected image, at least one of the following processes can be performed on the corrected Linear RGB image: cropping, color space conversion, gamma encoding, tone mapping, and other ISP processing, to obtain the final RGB image or video frame. The final RGB image or video frame can be displayed to the user on the screen of an electronic device.

[0182] The embodiments of this application are deployed in a separate processing module of the camera ISP, but can also be deployed in an image signal processor, neural network processor, graphics processing unit (GPU), central processing unit (CPU), field programmable gate array (FPGA), or application-specific integrated circuit. The embodiments of this application can be applied to security monitoring, night vision cameras, vehicle-mounted cameras, robot vision, outdoor low-light photography, and other imaging systems that require consideration of nighttime light intake, image detail, and color fidelity.

[0183] Based on the same inventive concept, corresponding to the above method embodiments, this application also provides a training apparatus for a parameter prediction network, such as... Figure 7 As shown, the device includes: an acquisition module 701, a prediction module 702, a correction module 703, a determination module 704, and an adjustment module 705;

[0184] The acquisition module 701 is used to acquire training samples and training labels. The training samples are images of the sample environment captured by the image acquisition device under visible light and near-infrared light conditions with short exposure. The training labels are images of the sample environment captured by the image acquisition device under near-infrared light cutoff conditions with long exposure. The illuminance of the sample environment is lower than the preset illuminance.

[0185] The prediction module 702 is used to predict the parameter correction map corresponding to the training sample through the parameter prediction network. The parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the training sample.

[0186] The correction module 703 is used to perform color correction on the training samples based on the parameter correction map to obtain a corrected image;

[0187] The determination module 704 is used to determine the image loss based on the corrected image and the training labels;

[0188] The adjustment module 705 is used to adjust the network parameters of the parameter prediction network using the image loss, and to call the prediction module to perform the step of predicting the parameter correction map corresponding to the training sample through the parameter prediction network until the parameter prediction network converges, thereby obtaining the trained parameter prediction network.

[0189] In some embodiments, the prediction module 702 is specifically used for:

[0190] The training samples are downsampled, and the downsampled images are input into the parameter prediction network.

[0191] The input data is feature extracted by the feature extraction module of the parameter prediction network, the input features are downsampled by the downsampling module of the parameter prediction network, and a parameter correction map of a specified dimension is output by the output layer of the parameter prediction network. The feature extraction module includes a convolutional layer and an activation layer. The height of the specified dimension is smaller than the height of the training sample, the width is smaller than the width of the training sample, and the number of channels is 9.

[0192] In some embodiments, the parameter correction map includes feature maps of 9 channels; the correction module 703 is specifically used for:

[0193] The resolution of the parameter correction map is converted to the resolution of the training sample; for each pixel in the training sample, a local color correction matrix is ​​constructed using the feature points of the corresponding 9 channels of the pixel to perform color correction on the pixel, resulting in a corrected image composed of the corrected pixels; wherein the position of the pixel in the training sample is the same as the position of the corresponding feature point in the feature map after resolution conversion; or...

[0194] For each pixel in the training sample, a local color correction matrix is ​​constructed using the feature points of the nine channels corresponding to the image region where the pixel is located. The pixel is then color-corrected to obtain a corrected image composed of the corrected pixels. The position of the image region in the training sample is the same as the position of the feature points corresponding to the image region in the feature map.

[0195] In some embodiments, the determining module 704 is specifically used for:

[0196] The color reconstruction loss is determined based on the color difference between pixels at the same position in the corrected image and the training label.

[0197] The structure preservation loss is determined based on the pixel mean and pixel standard deviation of the corrected image and the training label, as well as the pixel covariance between the corrected image and the training label.

[0198] The image loss is determined based on the color reconstruction loss and the structure preservation loss.

[0199] In some embodiments, the determining module 704 is specifically used for:

[0200] An additional loss is determined, which includes at least one of the following: a chromatic difference loss determined based on the chromatic difference between the corrected image and the training label; a spatial smoothing loss determined based on the difference between adjacent feature points in the intermediate feature map; a brightness preservation loss determined based on the brightness difference between the corrected image and the training sample; and a regularization loss determined based on the norm of the network layer weights of the parameter prediction network. The intermediate feature map is an intermediate result obtained by the parameter prediction network processing the training sample.

[0201] The weighted sum of the color reconstruction loss, the structure preservation loss, and the additional loss is taken as the image loss.

[0202] Based on the same inventive concept, and corresponding to the above method embodiments, this application also provides an image color correction device, such as... Figure 8 As shown, the device includes: an acquisition module 801, a prediction module 802, and a correction module 803;

[0203] The acquisition module 801 is used to acquire the image to be corrected, which is an image of the target environment captured by the image acquisition device under visible light and near-infrared light conditions with a first illuminance, where the first illuminance is less than a preset illuminance.

[0204] The prediction module 802 is used to predict the parameter correction map corresponding to the image to be corrected through the parameter prediction network. The parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the image to be corrected. The parameter prediction network is trained by the parameter prediction network training method provided in the embodiments of this application.

[0205] The correction module 803 is used to perform color correction on each image region included in the image to be corrected based on the parameter correction map to obtain a corrected image.

[0206] This application also provides an electronic device, such as... Figure 9 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0207] Memory 903 is used to store computer programs;

[0208] When the processor 901 executes the program stored in the memory 903, it implements the steps of the method in the above method embodiment.

[0209] The electronic device also includes an image sensor for acquiring raw images and sending them to a processor so that the processor can train a parameter prediction network or perform image color correction based on the raw images.

[0210] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0211] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0212] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0213] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0214] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the training method or image color correction method for any of the above-mentioned parameter prediction networks.

[0215] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the parameter prediction network training methods or image color correction methods in the above embodiments.

[0216] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0217] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0218] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0219] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A training method for a parameter prediction network, characterized in that, The method includes: Acquire training samples and training labels. The training samples are images of the sample environment captured by the image acquisition device with short exposure under visible and near-infrared light conditions. The training labels are images of the sample environment captured by the image acquisition device with long exposure under near-infrared light cutoff conditions. The illuminance of the sample environment is lower than a preset illuminance. The parameter correction map corresponding to the training sample is predicted by the parameter prediction network, and the parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the training sample. Based on the parameter correction map, the training samples are color corrected to obtain the corrected image; Based on the corrected image and the training labels, determine the image loss; The network parameters of the parameter prediction network are adjusted using the image loss, and the step of predicting the parameter correction map corresponding to the training sample through the parameter prediction network is returned until the parameter prediction network converges, thus obtaining the trained parameter prediction network.

2. The method according to claim 1, characterized in that, The step of predicting the parameter correction map corresponding to the training samples through a parameter prediction network includes: The training samples are downsampled, and the downsampled images are input into the parameter prediction network. The input data is feature extracted by the feature extraction module of the parameter prediction network, the input features are downsampled by the downsampling module of the parameter prediction network, and a parameter correction map of a specified dimension is output by the output layer of the parameter prediction network. The feature extraction module includes a convolutional layer and an activation layer. The height of the specified dimension is smaller than the height of the training sample, the width is smaller than the width of the training sample, and the number of channels is 9.

3. The method according to claim 1, characterized in that, The parameter correction map includes feature maps of 9 channels; the step of color correction of the training samples based on the parameter correction map to obtain a corrected image includes: The resolution of the parameter correction map is converted to the resolution of the training sample; for each pixel in the training sample, a local color correction matrix is ​​constructed using the feature points of the corresponding 9 channels of the pixel to perform color correction on the pixel, resulting in a corrected image composed of the corrected pixels; wherein the position of the pixel in the training sample is the same as the position of the corresponding feature point in the feature map after resolution conversion; or... For each pixel in the training sample, a local color correction matrix is ​​constructed using the feature points of the nine channels corresponding to the image region where the pixel is located. The pixel is then color-corrected to obtain a corrected image composed of the corrected pixels. The position of the image region in the training sample is the same as the position of the feature points corresponding to the image region in the feature map.

4. The method according to any one of claims 1-3, characterized in that, The step of determining image loss based on the corrected image and the training labels includes: The color reconstruction loss is determined based on the color difference between pixels at the same position in the corrected image and the training label. The structure preservation loss is determined based on the pixel mean and pixel standard deviation of the corrected image and the training label, as well as the pixel covariance between the corrected image and the training label. The image loss is determined based on the color reconstruction loss and the structure preservation loss.

5. The method according to claim 4, characterized in that, The process of determining the image loss based on the color reconstruction loss and the structure preservation loss includes: An additional loss is determined, which includes at least one of the following: a chromatic difference loss determined based on the chromatic difference between the corrected image and the training label; a spatial smoothing loss determined based on the difference between adjacent feature points in the intermediate feature map; a brightness preservation loss determined based on the brightness difference between the corrected image and the training sample; and a regularization loss determined based on the norm of the network layer weights of the parameter prediction network. The intermediate feature map is an intermediate result obtained by the parameter prediction network processing the training sample. The weighted sum of the color reconstruction loss, the structure preservation loss, and the additional loss is taken as the image loss.

6. An image color correction method, characterized in that, The method includes: Acquire an image to be corrected, which is an image of the target environment captured by an image acquisition device under visible and near-infrared light conditions; The parameter prediction network predicts the parameter correction map corresponding to the image to be corrected, wherein the parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the image to be corrected, and the parameter prediction network is trained by the method described in any one of claims 1-5. Based on the parameter correction map, color correction is performed on the image to be corrected to obtain a corrected image.

7. A training device for a parameter prediction network, characterized in that, The device includes: The acquisition module is used to acquire training samples and training labels. The training samples are images of the sample environment captured by the image acquisition device under visible and near-infrared light conditions with short exposure. The training labels are images of the sample environment captured by the image acquisition device under near-infrared light cutoff conditions with long exposure. The illuminance of the sample environment is lower than the preset illuminance. The prediction module is used to predict the parameter correction map corresponding to the training sample through the parameter prediction network. The parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the training sample. The correction module is used to perform color correction on the training samples based on the parameter correction map to obtain a corrected image; A determination module is used to determine the image loss based on the corrected image and the training labels; The adjustment module is used to adjust the network parameters of the parameter prediction network using the image loss, and to call the prediction module to perform the step of predicting the parameter correction map corresponding to the training sample through the parameter prediction network until the parameter prediction network converges, thus obtaining the trained parameter prediction network.

8. An image color correction device, characterized in that, The device includes: The acquisition module is used to acquire the image to be corrected, which is an image of the target environment captured by the image acquisition device under visible light and near-infrared light conditions with a first illuminance, where the first illuminance is less than a preset illuminance. A prediction module is used to predict a parameter correction map corresponding to the image to be corrected through a parameter prediction network. The parameter correction map represents the local color correction matrix corresponding to multiple image regions included in the image to be corrected. The parameter prediction network is trained by the method described in any one of claims 1-5. The correction module is used to perform color correction on each image region included in the image to be corrected based on the parameter correction map, so as to obtain a corrected image.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images