Training of an image enhancement model, image enhancement method, and electronic device

Abstract

The present application relates to the technical field of image processing, in particular to training of an image enhancement model, an image enhancement method and an electronic device, the method comprising: obtaining a reference image and a sample image pair of two consecutive frames collected by a binocular camera, the sample image pair of the first frame being collected by the binocular camera without near-infrared light compensation under a first illumination, and the sample image pair of the second frame being collected by the binocular camera with near-infrared light compensation under the first illumination; inputting the sample image pair of the first frame and the sample image pair of the second frame into an image enhancement model, to determine an alignment relationship of the sample image pair of the first frame by using the image enhancement model, and to perform image registration and image enhancement on the sample image pair of the second frame based on the alignment relationship, to obtain a predicted enhanced image; performing loss calculation based on the predicted enhanced image and the reference image, to update parameters of the image enhancement model, and to determine a target image enhancement model. The model can realize image enhancement processing in a low-illumination environment.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to the training of image enhancement models, image enhancement methods, and electronic devices. Background Technology

[0002] Due to sensor limitations, cameras can only capture a limited number of photons in low-light environments. Traditional image processing typically increases image brightness by increasing camera gain, which results in images containing a lot of noise and thus affecting imaging quality.

[0003] To address this issue, one approach is to increase the camera's exposure time, but this method is not well-suited for scenes with complex motion. Another approach is to use a flash to add light to the scene, but this can make the image look unnatural. Furthermore, using a flash can be visually distracting.

[0004] In recent years, the method of using near-infrared light to supplement illumination in low-light scenes has become increasingly popular. This is mainly because experimental studies have shown that the human eye is not sensitive to near-infrared light, so additional near-infrared illumination can be used to increase the number of photons received by the sensor, thereby obtaining a brighter and clearer image. However, since near-infrared light is in the invisible light category, the resulting image may have abnormal colors. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an image enhancement model training, an image enhancement method, and electronic depth to solve the image imaging problem in low-light scenes.

[0006] According to a first aspect, embodiments of the present invention provide a method for training an image enhancement model, comprising:

[0007] Acquire a reference image and a pair of two consecutive sample images captured by a binocular camera. The sample image pair includes an RGB image of a first field of view and a NIR image of a second field of view. The first sample image pair is acquired by the binocular camera under a first illuminance without near-infrared illumination, and the second sample image pair is acquired by the binocular camera under a first illuminance with near-infrared illumination.

[0008] The sample image pairs of the first frame and the sample image pairs of the second frame are input into the image enhancement model. The image enhancement model is used to determine the alignment relationship of the sample image pairs of the first frame and perform image registration and image enhancement on the sample image pairs of the second frame based on the alignment relationship to obtain the predicted enhanced image.

[0009] Loss calculations are performed based on the predicted enhanced image and the reference image to update the parameters of the image enhancement model and determine the target image enhancement model.

[0010] The training method of the image enhancement model provided by the embodiment of the present application can solve the problem of image alignment between two domains because the RGB and NIR images collected by the binocular camera belong to different modalities. Specifically, the first frame of sample image pairs is obtained without infrared light supplement, and the visible light and infrared light in the natural environment are captured. The intensity of the visible light and infrared light under the first illumination is weak, which can make the difference between the RGB image and the NIR image in brightness / clarity / noise small. The first frame of sample image pairs can obtain a more accurate alignment relationship. The alignment relationship is applied to the second frame of sample image pairs, and a more accurate field of view alignment processing can be realized, so that a target image enhancement model with good effect can be trained, and then image enhancement processing in a low-illumination environment can be realized.

[0011] In some embodiments, the inputting the first frame of sample image pairs and the second frame of sample image pairs into the image enhancement model to determine the alignment relationship of the first frame of sample image pairs by using the image enhancement model and to perform field of view alignment and image enhancement on the second frame of sample image pairs based on the alignment relationship to obtain a predicted enhanced image comprises:

[0012] extracting features of the luminance channel of the first frame of sample image pairs by using a feature extraction module in the image enhancement model and determining the alignment relationship based on the extracted features;

[0013] aligning the second frame of sample image pairs to a preset field of view based on the alignment relationship by using a registration module in the image enhancement model to obtain a registration image of the preset field of view, wherein the preset field of view includes the first field of view or the second field of view;

[0014] performing image enhancement processing on the registration image and the original image in the preset field of view in the second frame of sample image pairs by using an image enhancement reconstruction module in the image enhancement model to obtain the predicted enhanced image.

[0015] The training method of the image enhancement model provided by the embodiment of the present application can reduce the color difference of the input image by only extracting the features of the luminance channel to determine the alignment relationship.

[0016] In some embodiments, the extracting features of the luminance channel of the first frame of sample image pairs by using a feature extraction module in the image enhancement model and determining the alignment relationship based on the extracted features comprises:

[0017] The luminance channels of the sample image pair of the first frame are respectively subjected to feature extraction, to obtain corresponding first features and second features;

[0018] Optical flow field prediction is performed based on the first features and the second features, to obtain the alignment relationship.

[0019] The training method of the image enhancement model provided in the embodiment can greatly increase the accuracy of optical flow calculation.

[0020] In some embodiments, the feature extraction on the luminance channels of the sample image pair of the first frame to obtain the first features and the second features comprises:

[0021] The first feature extraction unit in the feature extraction module is used to extract the features of the luminance channels of the sample images in the first field of view in the sample images of the first frame, to obtain the first features.

[0022] The second feature extraction unit in the feature extraction module is used to extract the features of the luminance channels of the sample images in the second field of view in the sample images of the first frame, to obtain the second features, and the first feature extraction unit and the second feature extraction unit are of the same structure but different parameters.

[0023] The training method of the image enhancement model provided in the embodiment uses the first feature extraction unit and the second feature extraction unit to respectively extract the features of the luminance channels of the sample images in two fields of view, wherein the first feature extraction unit and the second feature extraction unit are of the same structure to ensure that the size of the extracted feature data can be suitable for the subsequent enhancement reconstruction module, and the parameters are different to be suitable for the two fields of view.

[0024] In some embodiments, the image enhancement processing on the registration image and the original image in the preset field of view in the sample image pair of the second frame by the enhancement reconstruction module in the image enhancement model to obtain the predicted enhancement image comprises:

[0025] The registration image and the original image in the preset field of view in the sample image pair of the second frame are input into the attention unit in the enhancement reconstruction module to obtain an attention map.

[0026] The attention map and the registration image are fused to obtain a processed registration image.

[0027] The processed registration image and the original image are input into the enhancement reconstruction unit in the enhancement reconstruction module to obtain the predicted enhancement image.

[0028] The training method of the image enhancement model provided by the embodiment of the application can reduce the color difference between the two images and ensure the reliability of the obtained attention map.

[0029] In some embodiments, the inputting of the registered image and the original image into the attention unit in the enhancement reconstruction module includes:

[0030] extracting features of a luminance channel of the registered image and the original image to obtain third features and fourth features;

[0031] connecting the third features and the fourth features to obtain connection features;

[0032] performing convolution and activation function processing on the connection features to obtain the attention map.

[0033] The training method of the image enhancement model provided by the embodiment of the application can reduce the color difference between the two images and ensure the reliability of the obtained attention map.

[0034] In some embodiments, the obtaining of the reference image includes:

[0035] obtaining an image collected by the binocular camera under a second illumination without infrared light compensation to obtain the reference image, the second illumination being greater than the first illumination;

[0036] or,

[0037] controlling the binocular camera to be in a long shutter mode and obtaining an image collected by the binocular camera under the first illumination without infrared light compensation to obtain the reference image.

[0038] According to a second aspect, the embodiment of the application further provides an image enhancement method, including:

[0039] obtaining a continuous two-frame image pair collected by a binocular camera, the image pair being an RGB image and an NIR image, the RGB image and the NIR image being located in different fields of view of the binocular camera, the first-frame image pair being collected by the binocular camera without near-infrared light compensation, and the second-frame image pair being collected by the binocular camera with near-infrared light compensation;

[0040] input the pair of the two continuous frames of images into a target image enhancement model to determine a target enhanced image, the target image enhancement model being trained according to the training method of the image enhancement model of the first aspect or any one of the embodiments of the first aspect.

[0041] The image enhancement method provided in the embodiments of the present application can perform image enhancement processing on a pair of two continuous frames of images collected by a binocular camera based on a target image enhancement model obtained, and can realize imaging in a low-illumination environment and improve low-illumination imaging quality.

[0042] According to a third aspect, the embodiments of the present application further provide a training device of an image enhancement model, comprising:

[0043] The first acquisition module is configured to acquire a reference image and a pair of two continuous frames of sample images collected by a binocular camera, the pair of sample images comprising an RGB image of a first field of view and an NIR image of a second field of view, wherein the first frame of sample images is collected by the binocular camera without near-infrared light compensation under a first illumination, and the second frame of sample images is collected by the binocular camera with near-infrared light compensation under the first illumination;

[0044] The prediction module is configured to input the first frame of sample images and the second frame of sample images into an image enhancement model to determine an alignment relationship of the first frame of sample images by using the image enhancement model and perform image registration and image enhancement on the second frame of sample images based on the alignment relationship, to obtain a predicted enhanced image.

[0045] The update module is configured to perform loss calculation based on the predicted enhanced image and the reference image to update parameters of the image enhancement model, and determine a target image enhancement model.

[0046] According to a fourth aspect, the embodiments of the present application further provide an image enhancement device, comprising:

[0047] The second acquisition module is configured to acquire a pair of two continuous frames of images collected by a binocular camera, the pair of images comprising an RGB image and an NIR image, the RGB image and the NIR image being located in different fields of view of the binocular camera, the first frame of images being collected by the binocular camera without near-infrared light compensation, and the second frame of images being collected by the binocular camera with near-infrared light compensation.

[0048] The enhancement module is configured to input the pair of two continuous frames of images into a target image enhancement model to determine a target enhanced image, the target image enhancement model being trained according to the training method of the image enhancement model of the first aspect or any one of the embodiments of the first aspect.

[0049] According to a fifth aspect, an electronic device is provided, comprising a memory and a processor, which are communicatively connected with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the training method of the image enhancement model according to the first aspect or any one of the implementations of the first aspect, or perform the image enhancement method according to the second aspect.

[0050] According to a sixth aspect, a computer readable storage medium is provided, which stores computer instructions for causing a computer to perform the training method of the image enhancement model according to the first aspect or any one of the implementations of the first aspect, or perform the image enhancement method according to the second aspect.

[0051] It should be noted that the corresponding beneficial effects of the training device of the image enhancement model, the image enhancement device, the electronic device and the computer readable storage medium provided by the embodiments of the present application are described above in the description of the training method of the image enhancement model or the image enhancement method, and will not be described here again. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0053] Figure 1 is a flowchart of the training method of the image enhancement model according to an embodiment of the present application;

[0054] Figure 2 is a flowchart of the training method of the image enhancement model according to an embodiment of the present application;

[0055] Figure 3 is a structural schematic diagram of the image enhancement model according to an embodiment of the present application;

[0056] Figure 4 is a structural schematic diagram of the enhancement reconstruction module according to an embodiment of the present application;

[0057] Figure 5 is a structural schematic diagram of the attention unit according to an embodiment of the present application;

[0058] Figure 6 is a flowchart of the image enhancement method according to an embodiment of the present application;

[0059] Figure 7is a structural block diagram of a training device of an image enhancement model according to an embodiment of the present application;

[0060] Figure 8 is a structural block diagram of an image enhancement device according to an embodiment of the present application;

[0061] Figure 9 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0063] The image enhancement method provided by the embodiments of the present application is a binocular infrared visible light fusion image enhancement method in a low-illumination environment, which aims to improve the camera imaging quality in a low-illumination environment and reconstruct an image with better definition, brightness, color and dynamic range. The image enhancement method is realized by relying on a target image enhancement model trained, and the target image enhancement model is trained by using two sample image pairs of consecutive frames collected by a binocular camera.

[0064] In the configuration of the binocular camera, an infrared (IR) filter is used for one of the cameras to only capture visible light to obtain an RGB image, and no IR filter is used for the other camera to simultaneously capture visible light and near-infrared light to obtain a near-infrared (NIR) image. The above-mentioned manner of obtaining the NIR image is only an example and does not limit the protection scope of the present application, and other manners can also be used to obtain the NIR image. Since in the configuration, the two cameras are located at different physical positions, it is necessary to perform dense registration, i.e. field of view alignment processing, on the images obtained by the binocular to obtain aligned images. The use of the aligned images for reconstruction makes the reconstructed image need to be able to fuse the color information of the RGB image and the detail texture information of the NIR image.

[0065] According to the embodiments of the present application, an image enhancement model training method or image enhancement method is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0066] The embodiment provides a training method of an image enhancement model, which can be applied to electronic devices such as computers and mobile terminals, Figure 1 The embodiment of the application provides a flowchart of the training method of the image enhancement model, as shown in Figure 1 The flowchart comprises the following steps:

[0067] S11, acquiring a reference image and a sample image pair of two continuous frames collected by a binocular camera.

[0068] The sample image pair comprises an RGB image of a first field of view and an NIR image of a second field of view, wherein the sample image pair of the first frame is collected by the binocular camera without near-infrared light compensation under a first illumination, and the sample image pair of the second frame is collected by the binocular camera with near-infrared light compensation under the first illumination.

[0069] The sample image pair of two continuous frames is realized by means of continuous twice snapshotting, the first snapshotting is performed without near-infrared light compensation, and the sample image pair of the first frame is obtained, the sample image pair of the first frame comprises RGB1 and NIR1; the second snapshotting is performed with near-infrared light compensation, and the sample image pair of the second frame is obtained, the sample image pair of the second frame comprises RGB2 and NIR2. Since the first snapshotting is performed without additional near-infrared light compensation, the gap between RGB1 and NIR1 can be greatly shortened. Since the interval between the two frames is very small, that is, the object moves very little between the two frames, and the sample image pair of the second frame is collected after near-infrared light compensation, the sample image pair of the second frame has good texture and other characteristics, so that the alignment relationship of the sample image pair of the first frame is applied to the sample image pair of the second frame for image registration and image enhancement, and good field of view alignment effect can be ensured.

[0070] The reference image is an expected output of a target image enhancement model obtained by training, and can be an image collected under normal illumination, or an image collected by using other manners and the like.

[0071] In some embodiments, the reference image is obtained by acquiring an image collected by the binocular camera without infrared light compensation under a second illumination, to obtain the reference image, and the second illumination is greater than the first illumination. The first illumination can be understood as the illumination in a low-illumination environment, and the second illumination can be understood as the illumination in a normal-illumination environment. For example, the illumination intensity in a normal environment is 100 lux, and the illumination lower than 10 lux can be regarded as low illumination, or other illumination thresholds can be set to determine the low illumination. In this case, no limitation is made, and the specific setting is made according to actual needs.

[0072] In some embodiments, the acquiring the reference image comprises: controlling the binocular camera to be in a long shutter mode, and acquiring an image collected by the binocular camera without infrared light compensation under the first illumination, to obtain the reference image. That is, the reference image is acquired by using the long shutter mode to collect the image under the first illumination.

[0073] S12, inputting the sample image pair of the first frame and the sample image pair of the second frame into the image enhancement model to determine the alignment relationship of the sample image pair of the first frame by using the image enhancement model, and performing image registration and image enhancement on the sample image pair of the second frame based on the alignment relationship, to obtain the predicted enhanced image.

[0074] Since the two images in the sample image pair are collected under different fields of view, field of view alignment is needed; meanwhile, color information of the RGB image and detail texture information of the NIR image need to be fused, so image enhancement is also needed on the basis of field of view alignment. Based on this, the image enhancement model comprises an alignment module for aligning the NIR image and the RGB image under stereovision, and a reconstruction enhancement module mainly for further fusing the aligned NIR image and RGB image, eliminating defects possibly caused in the alignment process, and reducing noise and improving brightness of the RGB image, to obtain an image with better definition, brightness, color, and dynamic range.

[0075] In the above, the alignment relationship is obtained based on the sample image pair of the first frame, and the alignment relationship is then applied to the sample image pair of the second frame to realize image registration of the sample image pair of the second frame; and then image enhancement is performed based on the result of the registration processing to obtain the predicted enhanced image.

[0076] S13, performing loss calculation based on the predicted enhanced image and the reference image to update parameters of the image enhancement model, to determine a target image enhancement model.

[0077] In the loss calculation, the difference between the predicted enhanced image and the reference image is used to calculate the loss function. The specific loss function is determined according to actual requirements, and is not limited herein. After multiple rounds of iteration, the parameters of the image enhancement model are updated to determine the target image enhancement model.

[0078] The training method of the image enhancement model provided in the embodiment can be used for an electronic device, such as a computer, a mobile terminal, and the like.

[0079] In the embodiment, a training method of an image enhancement model is provided, which can be used for an electronic device, such as a computer, a mobile terminal, and the like. Figure 2 is a flowchart of the training method of the image enhancement model according to the embodiment of the present application, as shown in the figure, the flow includes the following steps: Figure 2

[0080] S21, obtaining a reference image and a sample image pair of two consecutive frames collected by a binocular camera.

[0081] The sample image pair includes an RGB image of a first field of view and an NIR image of a second field of view, wherein the sample image pair of the first frame is collected by the binocular camera without near-infrared light compensation under a first illumination, and the sample image pair of the second frame is collected by the binocular camera with near-infrared light compensation under the first illumination.

[0082] For details, please refer to S11 of the embodiment shown in Figure 1 which will not be described here.

[0083] S22, inputting the sample image pair of the first frame and the sample image pair of the second frame into the image enhancement model, to determine an alignment relationship of the sample image pair of the first frame by using the image enhancement model, and to perform image registration and image enhancement on the sample image pair of the second frame based on the alignment relationship, to obtain a predicted enhanced image.

[0084] Specifically, the above S22 includes:

[0085] S221, performing feature extraction on a luminance channel of the sample image pair of the first frame by using a feature extraction module in the image enhancement model, and determining an alignment relationship based on the extracted features. ​

[0086] As mentioned above, the alignment module in the image enhancement model is used to achieve field-of-view alignment. Specifically, the alignment module includes a feature extraction module for feature extraction and a registration module for registration. For the sample image pair of the first frame, features are extracted from the brightness channels of the two images respectively to obtain the extracted features. Then, the alignment relationship is determined based on the two extracted features. The method for determining the alignment relationship includes, but is not limited to, calculating the similarity between the two features, or using the two features to predict the optical flow field.

[0087] In some embodiments, S221 includes:

[0088] (1) Extract features from the brightness channels of the sample image pairs in the first frame to obtain the corresponding first feature and second feature.

[0089] (2) Based on the first feature and the second feature, the optical flow field is predicted to obtain the alignment relationship.

[0090] like Figure 3 As shown, features are extracted from the brightness channels of the two images in the first frame's sample image pair to obtain the first feature and the second feature. Then, PWCNet units are used to predict the optical flow field using the first and second features, yielding the optical flow field, also known as the alignment relationship. Simultaneously, using the first frame's sample image pair to calculate the optical flow field and then applying it to the second frame's sample image pair can significantly increase the accuracy of optical flow calculation.

[0091] It should be noted that, Figure 3 as well as Figure 4 In this context, cam1 can be an image from either the first or second field of view. cam2 corresponds to a different field of view than cam1. When cam1 is an image from the first field of view, cam2 is an image from the second field of view; conversely, when cam1 is an image from the second field of view, cam2 is an image from the first field of view. The specific settings depend on actual needs and are not limited here.

[0092] In some embodiments, step (1) of S221 above includes:

[0093] 1.1) Using the first feature extraction unit in the feature extraction module, extract the features of the brightness channel of the sample image of the first field of view in the sample image of the first frame to obtain the first feature.

[0094] 1.2) Using the second feature extraction unit in the feature extraction module, extract the features of the brightness channel of the sample image of the second field of view in the sample image of the first frame to obtain the second feature. The first feature extraction unit and the second feature extraction unit have the same structure but different parameters.

[0095] For feature extraction of the luminance channel, the feature extraction of the luminance channel is performed using two first feature extraction units and a second feature extraction unit which are the same in structure but different in parameters, specifically, the two images of the sample image pair of the first frame are converted from the RGB space to the YUV space, and then features are extracted only on the Y channel to obtain the first features and the second features, which further reduces the color difference between the inputs.

[0096] The first feature extraction unit and the second feature extraction unit are used to perform feature extraction of the luminance channel on the sample images under the two fields of view, respectively, wherein the first feature extraction unit and the second feature extraction unit are the same in structure to ensure that the size of the extracted feature data can be suitable for the subsequent enhanced reconstruction module, and the parameters are different to be suitable for the two different fields of view.

[0097] S222, using a registration module in the image enhancement model, aligning the sample image pair of the second frame to the preset field of view based on the alignment relationship to obtain a registration image of the preset field of view.

[0098] The preset field of view includes the first field of view or the second field of view.

[0099] As described above, the sample image pair of the first frame is used to determine the alignment relationship, and then the alignment relationship is applied to the sample image pair of the second frame to realize field of view alignment, which can also be called image registration. When performing field of view alignment, the second field of view can be aligned to the first field of view, or the first field of view can be aligned to the second field of view, or further, the first field of view can be aligned to the second field of view, and the second field of view can be aligned to the first field of view, to obtain two registration images, and then the subsequent image reconstruction enhancement is performed respectively.

[0100] The processing principle of the registration module is that if the second field of view is aligned to the first field of view, the alignment relationship is applied to the original image under the second field of view in the sample image pair of the second frame, and the original image under the second field of view is processed into a registration image under the first field of view. In this case, the registration image under the first field of view and the original image under the first field of view in the sample image pair of the second frame are obtained.

[0101] If the first field of view is aligned to the second field of view, the alignment relationship is applied to the original image under the first field of view in the sample image pair of the second frame, and the original image under the first field of view is processed into a registration image under the second field of view. In this case, the registration image under the second field of view and the original image under the second field of view in the sample image pair of the second frame are obtained.

[0102] S223, performing image enhancement processing on the original image in the preset field of view in the registered image and the sample image pair of the second frame by using an enhancement reconstruction module in the image enhancement model, to obtain a predicted enhanced image.

[0103] The enhancement reconstruction module is used to realize the fusion of the RGB image and the NIR image, i.e., the fusion of the registered image and the original image. The goal of fusing the registered image and the original image is to extract edge and texture information from the NIR image, extract color information from the RGB image, and fuse them to generate an image with better noise reduction, clarity, color, and brightness. Therefore, the ideal fusion process should consider the characteristics of the image spatial structure, so that the reconstructed pixel intensity in the same image structure should be uniform. The U-net architecture is a pyramid structure composed of an encoder and a decoder, which enables the reconstructed pixel to have good perception of its adjacent pixels, and is very suitable for the purpose of visible-infrared light fusion. Based on this, in the embodiment, the U-net architecture is used to realize the enhancement reconstruction module. The registered image and the original image are input into the enhancement reconstruction module for image enhancement processing, to obtain a predicted enhanced image.

[0104] In some embodiments, S223 includes:

[0105] (1) inputting the registered image and the original image in the preset field of view in the sample image pair of the second frame into an attention unit in the image enhancement reconstruction module, to obtain an attention map.

[0106] (2) fusing the attention map and the registered image to obtain a processed registered image.

[0107] (3) inputting the processed registered image and the original image into an enhancement reconstruction unit in the image enhancement reconstruction module, to obtain a predicted enhanced image.

[0108] As shown above, the optical flow field is calculated using the sample image pair of the first frame in the first snapshot, and then applied to the sample image pair of the second frame in the second snapshot, which can greatly increase the accuracy of the optical flow calculation. However, due to the modal difference between the sample image pair of the first frame, and due to the existence of binocular disparity and the modal difference between the images in the sample image pair, occlusion may exist between the binocular images, which may cause the calculated optical flow field to still have inaccuracies, resulting in artifacts in the finally fused image. Based on this, the attention unit is used to suppress the structural inconsistencies that still exist after the alignment processing before the image enhancement processing, to guide the subsequent fusion reconstruction process.

[0109] For example, as shown in FIG. 6, the registered image (combined with the attention map) and the original image are input into the enhancement reconstruction module to obtain a predicted enhanced image. Figure 4 As shown in FIG. 6, the registered image (combined with the attention map) and the original image are input into the enhancement reconstruction module to obtain a predicted enhanced image. Figure 3At this time, the registration image is the registration of the original image of the first camera in the second frame to the image obtained in the field of view corresponding to the second camera, and the original image of the second camera in the second frame is input into the attention unit to obtain an attention map, and then the attention map is point multiplied with the registration image to suppress the part with inconsistent input structures, and finally the registration image and the original image of the second camera in the second frame are sent into the enhanced reconstruction unit to obtain a predicted enhanced image.

[0110] In some embodiments, step (1) of S223 comprises:

[0111] 1.1) Extracting features of the registration image and the luminance channel of the original image to obtain third features and fourth features.

[0112] 1.2) Connecting the third features and the fourth features to obtain connected features.

[0113] 1.3) Convolution and activation function processing of the connected features to obtain an attention map.

[0114] The principle of feature extraction of the luminance channel is as described above. When performing feature extraction of the luminance channel, since the convolutional neural network similar to U-Net can learn the depth features of the image, the features themselves will contain rich structural information. Therefore, in this embodiment, U-Net is used to construct the feature extraction part in the attention unit to achieve feature extraction of the luminance channel to obtain third features and fourth features.

[0115] As shown in Figure 5 , only the Y channel is used for feature extraction to obtain third features f1 and fourth features f2. Then f1 and f2 are connected (i.e., concatenated) together, and after multi-layer convolution and sigmoid activation function, the output is converted into a weight of [0, 1] to obtain an attention map.

[0116] By extracting only the features of the luminance channel of the registration image and the original image, the color difference between the two images can be reduced, ensuring the reliability of the obtained attention map.

[0117] S23, loss calculation based on the predicted enhanced image and the reference image to update the parameters of the image enhancement model to determine the target image enhancement model.

[0118] For details, please refer to S13 of the embodiment shown in Figure 1 , which will not be described here.

[0119] The training method of the image enhancement model provided in this embodiment only extracts features of the luminance channel to determine the alignment relationship, which can reduce the color difference between the input images.

[0120] The embodiment provides an image enhancement method, which can be used for electronic devices such as computers and mobile terminals, Figure 6 A flowchart of a training method of an image enhancement model according to an embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 6

[0121] S31, acquiring a pair of images of two continuous frames collected by a binocular camera.

[0122] The pair of images is an RGB image and an NIR image, the RGB image and the NIR image are located in different fields of view of the binocular camera, the pair of images of the first frame is collected by the binocular camera without near-infrared light compensation, and the pair of images of the second frame is collected by the binocular camera with near-infrared light compensation.

[0123] S32, inputting the pair of images of two continuous frames into a target image enhancement model to determine a target enhanced image.

[0124] The target image enhancement model is trained according to the training method of the image enhancement model in any of the above embodiments, and the structural details of the target image enhancement model are described above and will not be repeated here.

[0125] In the image enhancement processing, the images input into the target image enhancement model include a pair of images of two continuous frames, the pair of images of the first frame is collected without near-infrared light compensation and is used to determine the alignment relationship, and the alignment relationship is applied to the pair of images of the second frame and is subjected to reconstruction and enhancement processing to obtain the target enhanced image.

[0126] The image enhancement method provided in the embodiment can perform imaging in a low-illumination environment and improve the low-illumination imaging quality on the basis of the obtained target image enhancement model.

[0127] In the embodiment, a training device of an image enhancement model or an image enhancement device is also provided, which is used to implement the above embodiments and preferred embodiments, and will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is conceived.

[0128] The embodiment provides a training device of an image enhancement model, as shown in FIG. 2, which includes: Figure 7

[0129] ​​The first acquisition module 41 is configured to acquire a reference image and a sample image pair of two continuous frames collected by a binocular camera, the sample image pair comprising an RGB image of a first field of view and an NIR image of a second field of view, wherein the sample image pair of the first frame is collected by the binocular camera without near-infrared light compensation under a first illumination, and the sample image pair of the second frame is collected by the binocular camera with near-infrared light compensation under the first illumination;

[0130] The prediction module 42 is configured to input the sample image pair of the first frame and the sample image pair of the second frame into an image enhancement model, to determine an alignment relationship of the sample image pair of the first frame by using the image enhancement model, and to perform image registration and image enhancement on the sample image pair of the second frame based on the alignment relationship, to obtain a predicted enhanced image.

[0131] The update module 43 is configured to perform loss calculation based on the predicted enhanced image and the reference image, to update parameters of the image enhancement model, and to determine a target image enhancement model.

[0132] In some embodiments, the prediction module 42 comprises:

[0133] The extraction unit is configured to perform feature extraction on a luminance channel of the sample image pair of the first frame by using a feature extraction module in the image enhancement model, and to determine the alignment relationship based on the extracted features.

[0134] The alignment unit is configured to align the sample image pair of the second frame to a preset field of view based on the alignment relationship by using a registration module in the image enhancement model, to obtain a registration image of the preset field of view, and the preset field of view comprises the first field of view or the second field of view.

[0135] The enhancement reconstruction unit is configured to perform image enhancement processing on the registration image and an original image in the sample image pair of the second frame under the preset field of view by using an enhancement reconstruction module in the image enhancement model, to obtain the predicted enhanced image.

[0136] In some embodiments, the extraction unit comprises:

[0137] The extraction subunit is configured to perform feature extraction on the luminance channel of the sample image pair of the first frame respectively, to obtain corresponding first features and second features.

[0138] The optical flow field prediction subunit is configured to perform optical flow field prediction based on the first features and the second features, to obtain the alignment relationship.

[0139] In some embodiments, the extraction subunit comprises:

[0140] a first extraction subunit configured to extract, by using a first feature extraction unit in the feature extraction module, features of a luminance channel of a sample image in the first field of view in the sample image of the first frame, to obtain the first features;

[0141] a second extraction subunit configured to extract, by using a second feature extraction unit in the feature extraction module, features of a luminance channel of a sample image in the second field of view in the sample image of the first frame, to obtain the second features, the first feature extraction unit and the second feature extraction unit being of the same structure and different parameters.

[0142] In some embodiments, the enhancement reconstruction unit comprises:

[0143] an attention subunit configured to input the registration image and the original image in the preset field of view in the sample image pair of the second frame into an attention unit in the enhancement reconstruction module, to obtain an attention map;

[0144] a fusion subunit configured to fuse the attention map and the registration image, to obtain a processed registration image;

[0145] an enhancement subunit configured to input the processed registration image and the original image into an enhancement reconstruction unit in the enhancement reconstruction module, to obtain the predicted enhancement image.

[0146] In some embodiments, the attention subunit comprises:

[0147] a third extraction subunit configured to extract features of a luminance channel of the registration image and the original image, to obtain third features and fourth features;

[0148] a connection subunit configured to connect the third features and the fourth features to obtain connected features;

[0149] a processing subunit configured to perform convolution and activation function processing on the connected features, to obtain the attention map.

[0150] In some embodiments, the acquisition module comprises:

[0151] a first acquisition unit configured to acquire an image collected by the binocular camera under a second illumination without infrared fill light, to obtain the reference image, the second illumination being greater than the first illumination;

[0152] or,

[0153] a second acquisition unit configured to control the binocular camera to be in a long shutter mode, and acquire an image collected by the binocular camera under the first illumination without infrared fill light, to obtain the reference image.

[0154] The embodiment provides an image enhancement device, as shown in Figure 8 The embodiment provides an image enhancement device, as shown in

[0155] The second acquisition module 51 is configured to acquire a continuous two-frame image pair collected by a binocular camera, the image pair is an RGB image and an NIR image, the RGB image and the NIR image are located in different fields of view of the binocular camera, the first-frame image pair is collected by the binocular camera without near-infrared light compensation, and the second-frame image pair is collected by the binocular camera with near-infrared light compensation.

[0156] The enhancement module 52 is configured to input the continuous two-frame image pair into a target image enhancement model to determine a target enhanced image, and the target image enhancement model is obtained by training according to the training method of the image enhancement model in any one of the above embodiments.

[0157] The training device of the image enhancement model or the image enhancement device in the embodiment is in the form of a functional unit, and the unit refers to an ASIC circuit, a processor and a memory for executing one or more software or fixed programs, and / or other devices that can provide the above functions.

[0158] Further function descriptions of the above various modules are the same as those of the above corresponding embodiments, and will not be described here.

[0159] The embodiment of the present application also provides an electronic device having the training device of the image enhancement model shown in Figure 7 Or the image enhancement device shown in Figure 8 .

[0160] Please refer to Figure 9 , Figure 9 The embodiment of the present application also provides an electronic device having the training device of the image enhancement model shown in Figure 9 As shown in the structure schematic diagram of the electronic device, the electronic device can include at least one processor 61, such as a CPU (Central Processing Unit, central processor), at least one communication interface 63, a memory 64, and at least one communication bus 62. The communication bus 62 is used to realize the connection and communication between the components. The communication interface 63 can include a display screen (Display), a keyboard (Keyboard), and an optional communication interface 63, which can also include a standard wired interface and a wireless interface. The memory 64 can be a high-speed RAM memory (Random Access Memory, volatile random access memory), or a non-volatile memory, such as at least one disk memory. The memory 64 can also be at least one storage device located away from the aforementioned processor 61. The processor 61 can be combined with the communication interface 63 to form a communication processor (Communication Processor, CP). Figure 7or Figure 8 The described apparatus, the memory 64 stores an application program, and the processor 61 invokes the program code stored in the memory 64 for executing any of the above method steps.

[0161] The communication bus 62 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the figure to represent only one bus or only one type of bus.

[0162] The memory 64 can include a volatile memory (e.g., a random-access memory (RAM)), a non-volatile memory (e.g., a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD)), or a combination thereof.

[0163] The processor 61 can be a central processing unit (CPU), a network processor (NP), or a combination thereof.

[0164] The processor 61 can further include a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0165] Optionally, the memory 64 is further configured to store program instructions. The processor 61 can invoke the program instructions to implement the training method of the image enhancement model or the image enhancement method as shown in any embodiment of the present application.

[0166] The embodiment of the present application further provides a non-transitory computer storage medium, which stores computer executable instructions. The computer executable instructions can execute the training method of the image enhancement model or the image enhancement method in any method embodiment described above. The storage medium can be a disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory, a Hard Disk Drive (HDD) or a Solid-State Drive (SSD), etc. The storage medium can also include a combination of the above-mentioned storage devices.

[0167] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes are intended to fall within the scope of the appended claims.

Claims

1. A method for training an image enhancement model, characterized in that, include: Acquire a reference image and a pair of two consecutive sample images captured by a binocular camera. The sample image pair includes an RGB image of a first field of view and a NIR image of a second field of view. The first sample image pair is acquired by the binocular camera under a first illuminance without near-infrared illumination, and the second sample image pair is acquired by the binocular camera under a first illuminance with near-infrared illumination. The sample image pairs of the first frame and the sample image pairs of the second frame are input into the image enhancement model. The image enhancement model is used to determine the alignment relationship of the sample image pairs of the first frame and perform image registration and image enhancement on the sample image pairs of the second frame based on the alignment relationship to obtain the predicted enhanced image. Loss calculations are performed based on the predicted enhanced image and the reference image to update the parameters of the image enhancement model and determine the target image enhancement model.

2. The method according to claim 1, characterized in that, The step of inputting the sample image pairs of the first frame and the sample image pairs of the second frame into the image enhancement model, and using the image enhancement model to determine the alignment relationship of the sample image pairs of the first frame, and performing field-of-view alignment and image enhancement on the sample image pairs of the second frame based on the alignment relationship to obtain a predicted enhanced image, includes: The feature extraction module in the image enhancement model is used to extract features from the brightness channel of the sample image pair in the first frame, and the alignment relationship is determined based on the extracted features. Using the registration module in the image enhancement model, the sample image pairs of the second frame are aligned to a preset field of view based on the alignment relationship to obtain the registered image of the preset field of view, wherein the preset field of view includes the first field of view or the second field of view; Using the enhancement and reconstruction module in the image enhancement model, image enhancement processing is performed on the registered image and the original image under the preset field of view in the sample image pair of the second frame to obtain the predicted enhanced image.

3. The method according to claim 2, characterized in that, The step of using the feature extraction module in the image enhancement model to extract features from the brightness channel of the sample image pair in the first frame and determining the alignment relationship based on the extracted features includes: Feature extraction is performed on the brightness channel of the sample image pair in the first frame to obtain the corresponding first feature and second feature; The alignment relationship is obtained by predicting the optical flow field based on the first feature and the second feature.

4. The method according to claim 3, characterized in that, The step of extracting features from the brightness channel of the sample image pair of the first frame to obtain a first feature and a second feature includes: Using the first feature extraction unit in the feature extraction module, the luminance channel features of the sample image of the first field of view in the sample image of the first frame are extracted to obtain the first feature; Using the second feature extraction unit in the feature extraction module, the luminance channel features of the sample image of the second field of view in the sample image of the first frame are extracted to obtain the second feature. The first feature extraction unit and the second feature extraction unit have the same structure but different parameters.

5. The method according to claim 2, characterized in that, The step of using the enhancement and reconstruction module in the image enhancement model to perform image enhancement processing on the registered image and the original image under the preset field of view in the sample image pair of the second frame to obtain the predicted enhanced image includes: The registered image and the sample image of the second frame are compared with the original image under the preset field of view and input into the attention unit in the enhancement and reconstruction module to obtain the attention map; The attention map is fused with the registration image to obtain the processed registration image; The processed registration image and the original image are input into the enhancement and reconstruction unit in the enhancement and reconstruction module to obtain the predicted enhanced image.

6. The method according to claim 5, characterized in that, The step of inputting the registered image and the original image into the attention unit of the enhancement and reconstruction module to obtain an attention map includes: Features of the brightness channels of the registered image and the original image are extracted to obtain the third feature and the fourth feature; Connecting the third feature and the fourth feature yields a connection feature; The attention map is obtained by performing convolution and activation function processing on the connection features.

7. The method according to claim 1, characterized in that, The acquisition of the reference image includes: The reference image is obtained by acquiring an image captured by the binocular camera under a second illuminance without infrared supplementary lighting, wherein the second illuminance is greater than the first illuminance. or, The binocular camera is controlled to be in long shutter mode, and the image captured by the binocular camera under the first illumination without infrared supplementary lighting is obtained to obtain the reference image.

8. An image enhancement method, characterized in that, include: Acquire two consecutive image pairs captured by a stereo camera. The image pair consists of an RGB image and a NIR image, which are located in different fields of view of the stereo camera. The first image pair is acquired by the stereo camera without near-infrared illumination, and the second image pair is acquired by the stereo camera with near-infrared illumination. The two consecutive frames of images are input into the target image enhancement model to determine the target enhanced image. The target image enhancement model is trained using the image enhancement model training method according to any one of claims 1-7.

9. An electronic device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the training method of the image enhancement model according to any one of claims 1-7, or to perform the image enhancement method according to claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the training method of the image enhancement model according to any one of claims 1-7, or to perform the image enhancement method according to claim 8.

Images