Training of an image enhancement model, image enhancement method, and electronic device

Abstract

The present application relates to the technical field of image processing, in particular to training of an image enhancement model, an image enhancement method and an electronic device, the method comprising: obtaining a sample image group, the sample image group comprising a sample image collected by a binocular camera and a reference image, the sample image being an RGB image and an NIR image collected by the binocular camera after near-infrared light compensation under a first illumination, the reference image being an RGB image collected by the binocular camera under a second illumination, the first illumination being lower than the second illumination; inputting the RGB image and the NIR image into an image enhancement model to perform field-of-view alignment and enhancement reconstruction processing on the RGB image and the NIR image by using the image enhancement model to obtain an image enhancement prediction image; and performing loss calculation based on the image enhancement prediction image and the reference image to update parameters of the image enhancement model and determine a target image enhancement model. The method solves the imaging problem in a low-illumination environment.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to the training of image enhancement models, image enhancement methods, and electronic devices. Background Technology

[0002] High-quality imaging in low light is a challenging yet significant task. On the one hand, it plays a crucial role in many fields, such as surveillance, automotive driving, and smartphone photography. On the other hand, exposure and lighting in low-light environments result in extremely low signal-to-noise ratios (SNR) and a corresponding severe degradation in image quality. The most common approach to address this problem is to use supplemental lighting, but this leads to light pollution and glare for pedestrians and drivers, and inappropriate lighting in low light conditions can cause discomfort. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an image enhancement model training method, an image enhancement method, and an electronic device to solve the imaging problem in low-light environments.

[0004] According to a first aspect, embodiments of the present invention provide an image enhancement method, comprising:

[0005] A sample image set is obtained, which includes sample images acquired by a binocular camera and a reference image. The sample images are images acquired by the binocular camera after near-infrared supplemental lighting at a first illuminance. The sample images include RGB images and NIR images, which are located in different fields of view of the binocular camera. The reference image is an RGB image acquired by the binocular camera at a second illuminance, where the first illuminance is lower than the second illuminance.

[0006] The RGB image and the NIR image are input into an image enhancement model to perform field-of-view alignment and enhanced reconstruction on the RGB image and the NIR image using the image enhancement model to obtain an image enhancement prediction image. The field-of-view alignment is achieved by applying the feature similarity relationship between the RGB image and the NIR image in the luminance channel to the RGB image and / or the NIR image.

[0007] Loss calculations are performed based on the image enhancement prediction image and the reference image to update the parameters of the image enhancement model and determine the target image enhancement model.

[0008] The image enhancement model training method provided in this invention collects RGB and NIR images from a binocular camera under a first illumination level as sample images, and RGB images under a second illumination level as reference images. The sample images and reference images are combined to form a sample group. That is, the image enhancement model is trained by constructing a sample image group to obtain the target image enhancement model. At the same time, the field of view alignment during the training process is achieved by applying the feature similarity relationship on the brightness channel of the sample images to the sample images. Since calculating the feature similarity relationship between features on the brightness channel can remove the differences caused by color, applying it to RGB images and / or NIR images can ensure a good field of view alignment effect. Therefore, this image enhancement model realizes the RGB-NIR image fusion of the binocular camera, thereby solving the imaging problem in low-light environments.

[0009] In some embodiments, inputting the RGB image and the NIR image into an image enhancement model to perform field-of-view alignment on the RGB image and the NIR image using the image enhancement model includes:

[0010] The first reference feature and the second reference feature are obtained by using two feature extraction units in the image enhancement model to extract features in the brightness channel of the RGB image and the NIR image respectively;

[0011] The image enhancement model uses two semantic feature extraction units to extract semantic features from the RGB image and the NIR image respectively to obtain the first semantic feature and the second semantic feature. The feature extraction unit has the same structure as the semantic feature extraction unit but different parameters.

[0012] The image enhancement model is used to calculate the feature similarity relationship between the first reference feature and the second reference feature, and the first semantic feature and / or the second semantic feature are aligned under a preset field of view based on the feature similarity relationship to obtain aligned features to achieve the field of view alignment.

[0013] The image enhancement model training method provided in this embodiment of the invention uses the fact that NIR images have no color in all three channels, while RGB images contain color. Therefore, similarity calculation is performed only on the luminance channel to remove the differences caused by color. At the same time, different feature extraction units and semantic feature extraction units are used to perform similarity calculation and feature aggregation respectively to complete the binocular view alignment. This can overcome the difficulty of binocular alignment caused by the obvious color difference between RGB and NIR images.

[0014] In some implementations, the step of calculating the feature similarity relationship between the first reference feature and the second reference feature, and performing feature alignment of the first semantic feature and / or the second semantic feature under a preset field of view based on the feature similarity relationship to obtain aligned features to achieve the field of view alignment includes:

[0015] Calculate the similarity between the first reference feature and the second reference feature to obtain a similarity matrix;

[0016] The values ​​of the elements in the similarity matrix are mapped to values ​​within a preset range to determine the feature similarity relationship.

[0017] The image enhancement model training method provided in this embodiment of the invention may still have inaccuracies in field-of-view alignment, leading to artifacts in the final aggregated image. To address this, a similarity matrix is ​​obtained by first calculating the similarity between two reference features, and then the similarity matrix is ​​mapped to weight values ​​within a preset range to obtain feature similarity relationships. This process utilizes the attention mechanism to suppress structural inconsistencies that still exist after field-of-view alignment, thereby guiding the subsequent fusion and reconstruction process.

[0018] In some implementations, calculating the feature similarity relationship between the first reference feature and the second reference feature includes:

[0019] Obtain the target row containing the first pixel in the first reference feature of the RGB image;

[0020] Extract the second pixel of the row corresponding to the target row from the second reference feature of the NIR image;

[0021] Calculate the similarity between the first pixel and the second pixel to determine the feature similarity relationship.

[0022] The image enhancement model training method provided in this embodiment of the invention is based on the epipolar constraint of a dual-camera system, that is, the pixel coordinates of the same point in the binocular vision world are located in the same row in the images of the two cameras. Based on this, only the similarity between feature points in one view and feature points in the same row in another view is calculated, rather than all feature points in the other view, which greatly reduces the amount of computation.

[0023] In some embodiments, the alignment feature includes a first alignment feature corresponding to the first semantic feature and a second alignment feature corresponding to the second semantic feature. The image enhancement model includes a first enhancement and reconstruction unit and a second enhancement and reconstruction unit with the same structure but different parameters. The step of using the image enhancement model to perform field-of-view alignment and enhancement and reconstruction processing on the RGB image and the NIR image to obtain an image enhancement prediction image includes:

[0024] The first image enhancement prediction image is obtained by using the first enhancement and reconstruction unit to perform enhancement and reconstruction processing on the features after aggregating the first alignment features and the second semantic features.

[0025] The second enhancement and reconstruction unit is used to perform enhancement and reconstruction processing on the features after the aggregation of the second alignment feature and the first semantic feature to obtain the second image enhancement prediction image.

[0026] The image enhancement model training method provided in this embodiment of the invention cannot guarantee that every pixel in the image is accurately aligned after processing the viewpoint. This leads to different learning difficulties for the two paths mentioned above. Therefore, enhancing both paths simultaneously, rather than enhancing them separately, allows for better supervised learning. In order to fully utilize the complementary information provided by the two viewpoints of the stereo camera, enhancing and reconstructing the images in both the left and right fields of view simultaneously can combine the complementary information of the left and right images while maintaining full utilization of the information in the single image, thereby further improving the reconstruction performance.

[0027] In some implementations, the reference image includes a first reference image and a second reference image, the first reference image and the second reference image being located in different fields of view of the stereo camera. The step of performing loss calculation based on the image enhancement prediction image and the reference image to update the parameters of the image enhancement model and determine the target image enhancement model includes:

[0028] A first loss is calculated based on the first reference image and the first image enhancement prediction image to obtain the first loss;

[0029] A second loss is calculated based on the second reference image and the second image-enhanced prediction image to obtain the second loss.

[0030] The parameters of the image enhancement model are updated based on the first loss and the second loss to determine the target image enhancement model.

[0031] The image enhancement model training method provided in this embodiment of the invention performs loss calculations on the image enhancement prediction images under two fields of view, and then updates the parameters of the image enhancement model based on the losses under the two fields of view. This enables the image enhancement model to learn complementary information from the images under the two fields of view as well as information from the image under a single field of view during the training process, thus ensuring the reliability of the obtained target image enhancement model.

[0032] In some implementations, the step of performing loss calculation based on the image enhancement prediction image and the reference image to update the parameters of the image enhancement model and determine the target image enhancement model includes:

[0033] Edge loss is calculated by applying edge loss to the image enhancement prediction image and the reference image;

[0034] The color loss is calculated by examining the image enhancement prediction image and the reference image to obtain the color loss.

[0035] Based on the edge loss and the color loss, the parameters of the image enhancement model are updated to determine the target image enhancement model.

[0036] The image enhancement model training method provided in this embodiment of the invention combines edge loss and color loss for loss calculation, which can ensure the accuracy of the trained target image enhancement model.

[0037] According to a second aspect, embodiments of the present invention also provide an image enhancement method, comprising:

[0038] Acquire a first image and a second image under different fields of view captured by the target binocular camera. The first image is an RGB image, and the second image is an NIR image.

[0039] The first image and the second image are input into the target image enhancement model to obtain the target image. The target image enhancement model is trained by the image enhancement model training method described in the first aspect or any embodiment of the first aspect of the present invention.

[0040] The image enhancement method provided in this invention performs image enhancement processing on the first image and the second image based on the trained accurate target image enhancement model, which can solve the image imaging problem under low light conditions.

[0041] According to a third aspect, embodiments of the present invention provide a training apparatus for an image enhancement model, comprising:

[0042] The first acquisition module is used to acquire a sample image group, which includes sample images acquired by a binocular camera and a reference image. The sample images are images acquired by the binocular camera after near-infrared supplemental lighting under a first illuminance, and the reference image is an image acquired by the binocular camera under a second illuminance. The sample images include RGB images and NIR images, which are located in different fields of view of the binocular camera. The first illuminance is lower than the second illuminance.

[0043] The prediction module is used to input the RGB image and the NIR image into the image enhancement model, so as to use the image enhancement model to perform field alignment and enhancement reconstruction processing on the RGB image and the NIR image to obtain the image enhancement prediction image. The field alignment is achieved by applying the feature similarity relationship between the RGB image and the NIR image in the luminance channel to the RGB image and / or the NIR image.

[0044] An update module is used to perform loss calculation based on the image enhancement prediction image and the reference image to update the parameters of the image enhancement model and determine the target image enhancement model.

[0045] According to a fourth aspect, embodiments of the present invention also provide an image enhancement apparatus, comprising:

[0046] The second acquisition module is used to acquire a first image and a second image under different fields of view captured by the target binocular camera. The first image is an RGB image and the second image is an NIR image.

[0047] An image processing module is used to input the first image and the second image into a target image enhancement model to obtain a target image. The target image enhancement model is trained by the image enhancement model training method described in the first aspect or any embodiment of the first aspect of the present invention.

[0048] According to a fifth aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform a training method for an image enhancement model as described in the first aspect or any embodiment of the first aspect, or to perform an image enhancement method as described in the second aspect.

[0049] According to a sixth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing the computer to perform a training method for an image enhancement model as described in the first aspect or any embodiment of the first aspect, or to perform an image enhancement method as described in the second aspect.

[0050] It should be noted that the corresponding beneficial effects of the training device, image enhancement device, electronic device and computer-readable storage medium of the image enhancement model provided in the embodiments of the present invention can be found in the description of the corresponding beneficial effects of the image enhancement model training method or image enhancement method above, and will not be repeated here. Attached Figure Description

[0051] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is a flowchart of a training method for an image enhancement model according to an embodiment of the present invention;

[0053] Figure 2 This is a flowchart of a training method for an image enhancement model according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram illustrating the training of an image enhancement model according to an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of the alignment network according to an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the field-of-view alignment structure according to an embodiment of the present invention;

[0057] Figure 6 This is a flowchart of an image enhancement method according to an embodiment of the present invention;

[0058] Figure 7 This is a structural block diagram of a training device for an image enhancement model according to an embodiment of the present invention;

[0059] Figure 8 This is a structural block diagram of an image enhancement device according to an embodiment of the present invention;

[0060] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The image enhancement model training method provided in this invention is used to train a target image enhancement model suitable for imaging in low-light environments. This image enhancement model training method is based on a binocular camera. When the camera uses an infrared filter, it only acquires visible light for imaging, i.e., an RGB image. When the camera does not use an infrared filter, it can acquire both visible and infrared light for imaging, i.e., a NIR image.

[0063] The image enhancement method provided in this embodiment of the invention is based on the above-mentioned target image enhancement model. It is an image enhancement method for low-light environments based on binocular RGB-NIR dual-light fusion, thereby realizing imaging in low-light environments.

[0064] According to embodiments of the present invention, a training method for an image enhancement model and an embodiment of the image enhancement method are provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0065] This embodiment provides a training method for an image enhancement model, which can be used in electronic devices such as computers and mobile terminals. Figure 1 This is a flowchart of a training method for an image enhancement model according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0066] S11, Obtain the sample image group.

[0067] The sample image group includes sample images and reference images acquired by the binocular camera. The sample images are images acquired by the binocular camera after near-infrared supplemental lighting at a first illuminance. The sample images include RGB images and NIR images, which are located in different fields of view of the binocular camera. The reference image is an image acquired by the binocular camera at a second illuminance, where the first illuminance is lower than the second illuminance.

[0068] When training the image enhancement model, a set of sample images is required for each iteration. This set includes both sample images and a reference image. The sample images are obtained in low light conditions with simultaneous NIR illumination, while the reference image is obtained under normal ambient light conditions. Specifically, when obtaining the sample image set, ambient lighting is adjusted to low light conditions while simultaneously applying NIR illumination. The first camera in the binocular camera system uses an infrared filter (IR filter), while the second camera does not. Therefore, the first camera acquires RGB images, and the second camera acquires NIR images. The first camera has a first field of view, and the second camera has a second field of view.

[0069] Then, with normal ambient lighting, the RGB images acquired by the first and / or second cameras using IR filters are used as the reference images. In this way, the RGB images and NIR images under low-light conditions, together with the reference images under normal lighting conditions, constitute a sample image set.

[0070] It should be noted that the specific values ​​of the first and second illuminance are set according to actual needs, and no restrictions are imposed on them here. In the description of the embodiments of the present invention, the first illuminance corresponds to low illuminance, and the second illuminance corresponds to normal ambient illuminance. For example, if the light intensity in a normal environment is 100 lux, illuminance below 10 lux can be referred to as low illuminance, or other illuminance thresholds can be set to determine low illuminance. No limitations are imposed here; the specific values ​​are set according to actual needs.

[0071] S12, input the RGB image and NIR image into the image enhancement model, and use the image enhancement model to perform field alignment and enhancement reconstruction processing on the RGB image and NIR image to obtain the image enhancement prediction image.

[0072] The field alignment is achieved by applying the feature similarity relationship between the RGB image and the NIR image in the luminance channel to the RGB image and / or the NIR image.

[0073] The image enhancement model includes an alignment module for field-of-view alignment and an enhancement and reconstruction unit for enhanced reconstruction processing. The alignment module is used to achieve field-of-view alignment, transforming the two fields of view of the binocular camera into one. For example, it transforms the RGB image of the first field of view into the second field of view, or transforms the NIR image of the second field of view into the first field of view, and so on. That is, only when two sample images from different fields of view are transformed into the same field of view can subsequent image enhancement and reconstruction processing be performed. Specifically, during field-of-view transformation, features on the brightness channels of the RGB image and the NIR image are extracted to obtain feature similarity relationships. These relationships are then applied to the RGB image and the NIR image to obtain alignment relationships, thus achieving field-of-view alignment. In other words, the result of field-of-view alignment is obtained.

[0074] In some implementations, after obtaining the feature similarity relationship, the feature similarity relationship is first applied to the RGB image and / or NIR image to perform a field-of-view transformation, and then the semantic features of the transformed image are extracted to obtain the aligned processing result. Alternatively, the semantic features of the RGB image and / or NIR image are first extracted, and then the feature similarity relationship is applied to the semantic features to obtain the field-of-view alignment processing result.

[0075] After the alignment module, the two sample images still belong to their respective domains. That is, the RGB image remains an RGB image after being aligned from the first field of view to the second field of view, and the NIR image remains an NIR image after being aligned from the second field of view to the first field of view. In this embodiment, the goal is to fuse the RGB colors and NIR details to achieve imaging in low-light environments. Therefore, after the alignment module, enhancement and reconstruction processing is required to obtain an enhanced prediction image that fuses the RGB and NIR images.

[0076] The alignment module can be implemented by calculating the similarity between the RGB image and the NIR image to determine the alignment relationship, and then using this alignment relationship to achieve field-of-view transformation of the image under different viewing fields; alternatively, it can utilize the optical flow field between the RGB image and the NIR image, using the optical flow field as the alignment relationship, and then using this alignment relationship to achieve field-of-view transformation of the image under different viewing fields. Of course, the alignment module can also be implemented in other ways, and no limitations are imposed on it here.

[0077] The enhancement and reconstruction unit is used to perform enhanced reconstruction based on the output of the alignment module. The goal is for this unit to fuse color information from the RGB image and detail information from the NIR image. The enhancement and reconstruction unit can be implemented using supervised learning; here, a reference image is introduced for supervised learning.

[0078] S13, loss calculation is performed based on the image enhancement prediction image and the reference image to update the parameters of the image enhancement model and determine the target image enhancement model.

[0079] For an image augmentation model, the desired output image is the reference image. Based on this, a loss is calculated using the difference between the predicted augmented image and the reference image, and the parameters of the image augmentation model are updated based on the loss calculation result. When calculating the loss, other losses can be combined; for example, smooth loss can be used to constrain the alignment module, ensuring that adjacent pixels have similar weights, etc. The specific loss function used is set according to actual needs, and no restrictions are placed here. After multiple rounds of parameter updates to the image augmentation model, the final target image augmentation model is determined.

[0080] During inference, the target image enhancement model takes as input the first image captured by the first camera and the second image captured by the second camera in the binocular camera system, and outputs the target enhanced image.

[0081] The image enhancement model training method provided in this embodiment acquires RGB and NIR images of a binocular camera under a first illumination level as sample images and images under a second illumination level as reference images. The sample images and reference images are combined to form a sample group. That is, the image enhancement model is trained by constructing a sample image group to obtain the target image enhancement model. The image enhancement model is used to realize RGB-NIR image fusion of the binocular camera, thereby solving the imaging problem in low-light environments.

[0082] This embodiment provides a training method for an image enhancement model, which can be used in electronic devices such as computers and mobile terminals. Figure 2 This is a flowchart of a training method for an image enhancement model according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0083] S21, Obtain the sample image group.

[0084] The sample image group includes sample images and reference images acquired by the binocular camera. The sample images are images acquired by the binocular camera after near-infrared supplemental lighting at a first illuminance. The sample images include RGB images and NIR images, which are located in different fields of view of the binocular camera. The reference image is an image acquired by the binocular camera at a second illuminance, where the first illuminance is lower than the second illuminance.

[0085] Please see details Figure 1 S11 of the illustrated embodiment will not be described again here.

[0086] S22, input the RGB image and NIR image into the image enhancement model, and use the image enhancement model to perform field alignment and enhancement reconstruction processing on the RGB image and NIR image to obtain the image enhancement prediction image.

[0087] The field alignment is achieved by applying the feature similarity relationship between the RGB image and the NIR image in the luminance channel to the RGB image and / or the NIR image.

[0088] Specifically, S22 includes:

[0089] S221, using two feature extraction units in the image enhancement model to extract features on the brightness channel of the RGB image and the NIR image to obtain the first reference feature and the second reference feature.

[0090] Two feature extraction units correspond to the first and second fields of view, respectively. These two units extract features from the luminance channels of the RGB and NIR images, overcoming the difficulties caused by the significant color differences between RGB and NIR images that hinder binocular alignment. The two feature extraction units have the same structure but different parameters. Since the processing procedures of the two units are similar, only the processing procedure of the feature extraction unit used for the RGB image will be described below.

[0091] When using the feature extraction unit to extract features from an RGB image, the image format of the RGB image is first converted to YUV space so that feature extraction can be performed only on the luminance channel. Since the three channels of a NIR image are all the same and have no color, while an RGB image contains color, feature extraction and similarity calculation are performed only on the Y channel to remove the differences caused by color.

[0092] For ease of description below, the features obtained by extracting the luminance channel features from the RGB image are referred to as the first reference features, and the features obtained by extracting the luminance channel features from the NIR image are referred to as the second reference features.

[0093] S222, using two semantic feature extraction units in the image enhancement model to extract semantic features from the RGB image and the NIR image respectively to obtain the first semantic feature and the second semantic feature.

[0094] The feature extraction unit has the same structure as the semantic feature extraction unit but different parameters.

[0095] The semantic feature extraction units also correspond to the first and second fields of view, respectively. That is, different semantic feature extraction units are used to extract semantic features from images in two different fields of view. The two semantic feature extraction units have the same structure but different parameters.

[0096] Both the feature extraction unit and the semantic feature extraction unit can be implemented based on convolutional neural networks. For convolutional neural networks, different depths correspond to different levels of semantic features. Shallow networks have high resolution and learn more detailed features, while deep networks have low resolution and learn more semantic features.

[0097] In some implementations, the semantic feature extraction unit is implemented using multiple residual layers, with the resolution reduced by half at each subsequent layer while the number of channels increases, to extract features with different semantic information. Two semantic feature extraction units are used to extract semantic features from the RGB image and the NIR image respectively, resulting in a first semantic feature in a first field of view and a second semantic feature in a second field of view.

[0098] It should be noted that the feature extraction unit and the semantic feature extraction unit are processed in parallel. The feature extraction unit is used to extract the features of the Y channel of the sample image and calculate the similarity between the first reference feature and the second reference feature to obtain the feature similarity relationship. The semantic feature extraction unit is used to extract the semantic features of the sample image and apply the feature similarity relationship to the semantic features to achieve field of view alignment.

[0099] In some implementations, the feature extraction unit and the semantic feature extraction unit have the same structure but different parameters, and both can be implemented using convolutional neural networks, differing only in their network parameters. The feature aggregation unit aims to use the similarity relationship calculated from the features on the luminance channel extracted by the feature extraction unit to fuse the semantic features of RGB and NIR. This operation needs to be performed on feature maps of the same shape, so the structure of the feature extraction unit and the semantic feature extraction unit must be consistent.

[0100] S223, calculate the feature similarity relationship between the first reference feature and the second reference feature using an image enhancement model, and perform feature alignment of the first semantic feature and / or the second semantic feature under a preset field of view based on the feature similarity relationship to obtain aligned features to achieve field of view alignment.

[0101] The preset field of view is the first field of view and / or the second field of view.

[0102] Similarity is calculated between the first reference feature and the second reference feature to obtain the feature similarity relationship between the features extracted in the brightness channel. The similarity is calculated using the dot product between vectors, specifically the dot product between the first and second reference features.

[0103] The similarity calculation involves calculating the similarity between each pair of feature points of the first reference feature and the second reference feature to construct the spatial relationship between the features of the two brightness channels, i.e., the feature similarity relationship.

[0104] In some implementations, calculating the feature similarity relationship between the first reference feature and the second reference feature includes:

[0105] (1) Obtain the target row where the first pixel point is located in the first reference feature of the RGB image.

[0106] (2) Extract the second pixel of the row corresponding to the target row from the second reference feature of the NIR image.

[0107] (3) Calculate the similarity between the first pixel and the second pixel to determine the feature similarity relationship.

[0108] Based on the epipolar constraint of the dual-camera system, meaning that the pixel coordinates of the same point in the binocular vision world are located in the same row in the images of the two cameras, the similarity is calculated only between feature points in one view and feature points in the same row in the other view, rather than all feature points in the other view, greatly reducing the computational load. Specifically, the target row where each first pixel point in the first reference feature is located is first determined; then, all second pixels in the row corresponding to the target row are extracted from the second reference feature, and the dot product between the first pixel point and all second pixels is calculated to determine the feature similarity relationship. For example, for the first pixel point with coordinates (1, 1), all second pixels in the first row are extracted from the second reference feature, and the dot product between the first pixel point and all second pixels is calculated to determine the feature similarity relationship.

[0109] If the preset field of view is the first field of view, then the second semantic features of the second field of view are aligned to the first field of view using feature similarity relationships to obtain the aligned second semantic features. The second semantic features obtained after the alignment process can also be called the second aligned features.

[0110] If the preset field of view is the second field of view, that is, the first semantic feature of the first field of view is aligned to the second field of view using the feature similarity relationship, and the first semantic feature after alignment is obtained. The first semantic feature obtained after the alignment can also be called the first aligned feature.

[0111] If the preset fields of view are a first field of view and a second field of view, then it means that the first field of view needs to be aligned to the second field of view, and the second field of view needs to be aligned to the first field of view. The first semantic feature of the first field of view is aligned to the second field of view using feature similarity relationships, resulting in the aligned first semantic feature, i.e., the first aligned feature; the second semantic feature of the second field of view is aligned to the first field of view using feature similarity relationships, resulting in the aligned second semantic feature, i.e., the second aligned feature.

[0112] In some implementations... Figure 3 A schematic diagram of the training structure of the image enhancement model is shown. For example... Figure 3As shown, the feature extraction unit includes a feature extraction unit for extracting the Y channel and a semantic feature extraction unit for extracting semantic features. The feature extraction unit is a multi-layer feature extraction unit, and the semantic feature extraction unit is a multi-layer semantic feature extraction unit. There are two feature extraction units, corresponding to the first field of view and the second field of view, respectively; similarly, there are two semantic feature extraction units, corresponding to the first field of view and the second field of view, respectively. Figure 3 Multi-layer semantic feature extraction, multi-layer feature extraction, and similarity calculation all belong to the alignment module.

[0113] In some implementations... Figure 4 The specific implementation of the alignment module is shown. Both multi-layer semantic feature extraction and multi-layer extraction are implemented based on residual units.

[0114] In some embodiments, S223 includes:

[0115] (1) Calculate the similarity between the first reference feature and the second reference feature to obtain the similarity matrix.

[0116] (2) Map the elements in the similarity matrix to values ​​within a preset range to determine the feature similarity relationship.

[0117] like Figure 5 As shown, the similarity between the first and second reference features is first calculated to obtain a similarity matrix. Then, the element values ​​in the similarity matrix are mapped to values ​​within a preset range to obtain the feature similarity relationship. The preset range can be [0, 1] or other ranges, without any restrictions. For example, the mapping can be implemented using the softmax function or other methods. This feature similarity relationship represents the similarity weight of a pixel in one viewpoint to the corresponding pixel in another viewpoint. This similarity weight is then used to align the semantic features of the other viewpoint to the current viewpoint. For example, if the first field of view is converted to a second field of view, the product of the first semantic feature and the feature similarity relationship is calculated to obtain the first semantic feature in the second field of view.

[0118] Based on this, the first semantic feature and / or the second semantic feature are aligned under a preset field of view based on feature similarity relationships to obtain aligned features to achieve the field of view alignment. For example, by using feature similarity relationships, the first semantic feature under the first field of view is aligned to the second field of view to obtain the first aligned feature corresponding to the first semantic feature; and / or, the second semantic feature under the second field of view is aligned to the first field of view to obtain the second aligned feature corresponding to the second semantic feature.

[0119] Inaccuracies may still exist in the field-of-view alignment, leading to artifacts in the final aggregated image. To address this, a similarity matrix is ​​first calculated between two reference features. This matrix is ​​then mapped to weight values ​​within a preset range to obtain feature similarity relationships. This process utilizes an attention mechanism to suppress structural inconsistencies that may still exist after field-of-view alignment, thereby guiding the subsequent fusion and reconstruction process.

[0120] S224. The image enhancement model uses the enhancement reconstruction unit to enhance and reconstruct the alignment features and corresponding semantic features to obtain the image enhancement prediction image.

[0121] The enhancement and reconstruction unit performs enhancement and reconstruction processing based on the alignment unit, and the output of the enhancement and reconstruction unit is the image enhancement prediction image.

[0122] In some embodiments, the alignment feature includes a first alignment feature corresponding to a first semantic feature and a second alignment feature corresponding to a second semantic feature, and the enhancement reconstruction unit includes a first enhancement reconstruction unit and a second enhancement reconstruction unit with the same structure but different parameters. Based on this, S224 above includes:

[0123] (1) The first image enhancement prediction image is obtained by using the first enhancement reconstruction unit to perform enhancement reconstruction processing on the features after the first alignment feature and the second semantic feature are aggregated.

[0124] (2) The second image enhancement prediction image is obtained by using the second enhancement reconstruction unit to perform enhancement reconstruction processing on the features after the second alignment feature and the first semantic feature are aggregated.

[0125] Viewpoint alignment processing cannot guarantee that every pixel in the image is accurately aligned, which leads to different learning difficulties for the two paths mentioned above. Therefore, enhancing both paths simultaneously, rather than enhancing them separately, allows for better supervised learning. To fully utilize the complementary information provided by the two viewpoints of the stereo camera, enhancing and reconstructing images from both left and right fields of view simultaneously can combine the complementary information from the left and right images while maintaining full utilization of information from a single image, thereby further improving reconstruction performance.

[0126] First, the first alignment feature and the second semantic feature under the same field of view are aggregated to obtain the first aggregated feature, and the second alignment feature and the first semantic feature under the same field of view are aggregated to obtain the second aggregated feature. Then, the first aggregated feature is enhanced and reconstructed using the first enhancement and reconstruction unit to obtain the first image enhancement prediction model, and the second aggregated feature is enhanced and reconstructed using the second enhancement and reconstruction unit to obtain the second image enhancement prediction model. The feature aggregation method can be concatenation, splicing, etc., and no limitation is made here.

[0127] S23, loss calculation is performed based on the image enhancement prediction image and the reference image to update the parameters of the image enhancement model and determine the target image enhancement model.

[0128] Specifically, S23 above includes:

[0129] S231, calculate the edge loss for the image enhancement prediction image and the reference image to obtain the edge loss.

[0130] For edge loss, the Sobel edge detection operator is used to extract the edges of the image enhancement prediction image and the reference image respectively, and then the difference is calculated, so that the contribution of edge loss during training comes only from the edges.

[0131] S232, calculate the color loss for the image enhancement prediction image and the reference image to obtain the color loss.

[0132] For color loss, a color histogram is used to measure color differences. A color histogram primarily describes the proportion of different colors in the entire image, without considering the spatial location of colors. Therefore, for an image, the resulting color histogram is a 3×256 matrix, where 3 corresponds to the three color channels (R, G, and B), and 256 represents the range of pixel values. For example, the color loss l can be calculated using the following formula. c :

[0133]

[0134] Among them, C pre As the reference image, C q Enhance the predicted image for the image.

[0135] S233, based on edge loss and color loss, updates the parameters of the image enhancement model to determine the target image enhancement model.

[0136] In addition to edge loss and color loss, a smooth loss can be used to constrain the attention map obtained by the alignment module, so that adjacent pixels have similar weights. Alternatively, other losses can be included, which can be set according to actual needs without any restrictions.

[0137] In some embodiments, the reference image includes a first reference image under the first field of view and a second reference image under the second field of view, and S23 includes:

[0138] (1) The first loss is calculated based on the first reference image and the first image enhancement prediction image to obtain the first loss.

[0139] (2) The second loss is calculated based on the second reference image and the second image enhancement prediction image to obtain the second loss.

[0140] (3) Update the parameters of the image enhancement model based on the first loss and the second loss to determine the target image enhancement model.

[0141] Similarly, the reference image includes images from two fields of view: a first reference image from a first field of view and a second reference image from a second field of view. A first loss is calculated using both the first reference image and the first image-enhanced prediction image; correspondingly, a second loss is calculated using both the second reference image and the second image-enhanced prediction image. Parameters are then updated based on the first and second losses to determine the target image enhancement model. The first loss includes, but is not limited to, the edge loss and color loss described above, and the second loss includes, but is not limited to, the edge loss and color loss described above.

[0142] After performing viewpoint alignment, it cannot be guaranteed that every pixel in the image is accurately aligned. This will result in different learning difficulties for the two paths mentioned above. Therefore, enhancing both paths simultaneously, rather than enhancing them separately, can better supervise learning.

[0143] For details on loss calculation, please refer to the relevant description in S23 above, which will not be repeated here.

[0144] The image enhancement model training method provided in this embodiment addresses the issue that NIR images have no color across all three channels, while RGB images contain color. Therefore, similarity calculation is performed only on the Y channel to remove color-related differences. Simultaneously, two different feature extraction modules and a semantic extraction module are used for similarity calculation and feature aggregation respectively to achieve binocular view alignment. This overcomes the difficulty of binocular alignment caused by the significant color differences between RGB and NIR images. Combining edge loss and color loss calculations ensures the accuracy of the trained target image enhancement model.

[0145] This embodiment provides an image enhancement method that can be used in electronic devices, such as computers and mobile terminals. Figure 6 This is a flowchart of an image enhancement method according to an embodiment of the present invention, such as... Figure 6 As shown, the process includes the following steps:

[0146] S31, acquire the first image and the second image under different fields of view captured by the target binocular camera.

[0147] The first image is an RGB image, and the second image is an NIR image.

[0148] S32, input the first image and the second image into the target image enhancement model to obtain the target image.

[0149] The target image enhancement model is trained according to the image enhancement model training method described above.

[0150] When using the target image enhancement model for inference, the first and second images captured by the target binocular camera under different fields of view are input into the target image enhancement model. After processing by the target image enhancement model, the target image can be obtained. The specific implementation process and principles of the target image enhancement model are described above and will not be repeated here.

[0151] The image enhancement method provided in this embodiment performs image enhancement processing on the first image and the second image based on the trained accurate target image enhancement model, which can solve the image imaging problem under low light conditions.

[0152] This embodiment also provides a training device for an image enhancement model and an image enhancement device, which are used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0153] This embodiment provides a training device for an image enhancement model, such as... Figure 7 As shown, it includes:

[0154] The first acquisition module 41 is used to acquire a sample image group, which includes sample images acquired by a binocular camera and a reference image. The sample images are images acquired by the binocular camera after near-infrared supplementary lighting under a first illuminance. The sample images include RGB images and NIR images, which are located in different fields of view of the binocular camera. The reference image is an image acquired by the binocular camera under a second illuminance, where the first illuminance is lower than the second illuminance.

[0155] The prediction module 42 is used to input the RGB image and the NIR image into the image enhancement model, so as to use the image enhancement model to perform field alignment and enhancement reconstruction processing on the RGB image and the NIR image to obtain the image enhancement prediction image. The field alignment is achieved by applying the feature similarity relationship between the RGB image and the NIR image in the brightness channel to the RGB image and / or the NIR image.

[0156] The update module 43 is used to perform loss calculation based on the image enhancement prediction image and the reference image to update the parameters of the image enhancement model and determine the target image enhancement model.

[0157] In some implementations, the prediction module 42 includes:

[0158] The first extraction unit is used to extract features in the brightness channel of the RGB image and the NIR image respectively using the two feature extraction units in the image enhancement model to obtain the first reference feature and the second reference feature;

[0159] The second extraction unit is used to extract semantic features from the RGB image and the NIR image respectively using the two semantic feature extraction units in the image enhancement model to obtain the first semantic feature and the second semantic feature. The feature extraction unit has the same structure as the semantic feature extraction unit but different parameters.

[0160] The feature alignment unit is used to calculate the feature similarity relationship between the first reference feature and the second reference feature using the image enhancement model, and to perform feature alignment of the first semantic feature and / or the second semantic feature under a preset field of view based on the feature similarity relationship to obtain aligned features to achieve the field of view alignment.

[0161] In some implementations, the feature alignment unit includes:

[0162] The first calculation subunit is used to calculate the similarity between the first reference feature and the second reference feature to obtain a similarity matrix;

[0163] The mapping subunit is used to map the values ​​of elements in the similarity matrix to values ​​within a preset range to determine the feature similarity relationship.

[0164] In some implementations, the first extraction unit includes:

[0165] The first acquisition subunit is used to acquire the target row where the first pixel point is located in the first reference feature of the RGB image;

[0166] An extraction subunit is used to extract the second pixel of the row corresponding to the target row from the second reference feature of the NIR image;

[0167] The second calculation subunit is used to calculate the similarity between the first pixel and the second pixel to determine the feature similarity relationship.

[0168] In some embodiments, the alignment feature includes a first alignment feature corresponding to the first semantic feature and a second alignment feature corresponding to the second semantic feature, and the enhancement reconstruction unit includes a first enhancement reconstruction unit and a second enhancement reconstruction unit with the same structure but different parameters. The enhancement reconstruction unit includes:

[0169] The first enhancement and reconstruction subunit is used to perform enhancement and reconstruction processing on the features after aggregating the first alignment features and the second semantic features using the first enhancement and reconstruction unit to obtain the first image enhancement prediction image.

[0170] The second enhancement and reconstruction subunit is used to perform enhancement and reconstruction processing on the features after the second alignment feature and the first semantic feature are aggregated, so as to obtain the second image enhancement prediction image.

[0171] In some embodiments, the reference image includes a first reference image and a second reference image, the first reference image and the second reference image being located in different fields of view of the binocular camera, and the update module 43 includes:

[0172] The first loss unit is used to perform a first loss calculation based on the first reference image and the first image enhancement prediction image to obtain a first loss.

[0173] The second loss unit is used to perform a second loss calculation based on the second reference image and the second image enhancement prediction image to obtain the second loss.

[0174] An update unit is used to update the parameters of the image enhancement model based on the first loss and the second loss, and to determine the target image enhancement model.

[0175] In some implementations, the updating unit includes:

[0176] The first calculation subunit is used to calculate the edge loss of the image enhancement prediction image and the reference image to obtain the edge loss;

[0177] The second calculation subunit is used to calculate the color loss of the image enhancement prediction image and the reference image to obtain the color loss;

[0178] An update subunit is used to update the parameters of the image enhancement model based on the edge loss and the color loss, and to determine the target image enhancement model.

[0179] This embodiment provides an image enhancement device, such as... Figure 8 As shown, it includes:

[0180] The second acquisition module 51 is used to acquire a first image and a second image under different fields of view captured by the target binocular camera. The first image is an RGB image and the second image is an NIR image.

[0181] Image processing module 52 is used to input the first image and the second image into the target image enhancement model to obtain the target image. The target image enhancement model is trained according to the above-described image enhancement model training method.

[0182] In this embodiment, the training device and the image enhancement device of the image enhancement model are presented in the form of functional units. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0183] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0184] This invention also provides an electronic device having the above-described features. Figure 7 The training device for the image enhancement model shown, or Figure 8 The image enhancement device shown.

[0185] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 9 As shown, the electronic device may include: at least one processor 61, such as a CPU (Central Processing Unit), at least one communication interface 63, memory 64, and at least one communication bus 62. The communication bus 62 is used to enable communication between these components. The communication interface 63 may include a display screen or a keyboard; optionally, the communication interface 63 may also include a standard wired interface or a wireless interface. The memory 64 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 64 may also be at least one storage device located remotely from the aforementioned processor 61. The processor 61 may be combined with... Figure 7 or Figure 8 The described apparatus has an application program stored in memory 64, and the processor 61 calls the program code stored in memory 64 to perform any of the above method steps.

[0186] The communication bus 62 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0187] The memory 64 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 64 may also include a combination of the above types of memory.

[0188] The processor 61 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0189] The processor 61 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0190] Optionally, memory 64 is also used to store program instructions. Processor 61 can invoke program instructions to implement the training method of the image enhancement model as shown in any embodiment of this application, or the image enhancement method as shown in any embodiment of this application.

[0191] This invention also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the training method or image enhancement method of the image enhancement model in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0192] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for training an image enhancement model, characterized in that, The method comprises the following steps: acquiring a sample image group, the sample image group comprising a sample image collected by a binocular camera and a reference image, the sample image being an image collected by the binocular camera after near-infrared light compensation under a first illumination, the sample image comprising an RGB image and an NIR image, the RGB image and the NIR image being located in different fields of view of the binocular camera, and the reference image being an RGB image collected by the binocular camera under a second illumination, the first illumination being lower than the second illumination; inputting the RGB image and the NIR image into an image enhancement model to perform field of view alignment and enhancement reconstruction processing on the RGB image and the NIR image by using the image enhancement model, to obtain an image enhancement prediction image, wherein the field of view alignment is achieved by applying a feature similarity relationship in a brightness channel of the RGB image and the NIR image to the RGB image and / or the NIR image; performing loss calculation based on the image enhancement prediction image and the reference image to update parameters of the image enhancement model, to determine a target image enhancement model.

2. The method of claim 1, wherein, The step of inputting the RGB image and the NIR image into the image enhancement model to perform field of view alignment by using the image enhancement model comprises the following steps: performing feature extraction in a brightness channel of the RGB image and the NIR image by using two feature extraction units in the image enhancement model to obtain first reference features and second reference features; performing semantic feature extraction on the RGB image and the NIR image by using two semantic feature extraction units in the image enhancement model to obtain first semantic features and second semantic features, the feature extraction units and the semantic feature extraction units being the same in structure and different in parameters; calculating a feature similarity relationship between the first reference features and the second reference features by using the image enhancement model, and performing feature alignment in a preset field of view on the first semantic features and / or the second semantic features based on the feature similarity relationship to obtain aligned features, to achieve the field of view alignment.

3. The method of claim 2, wherein, The step of calculating a feature similarity relationship between the first reference features and the second reference features, and performing feature alignment in a preset field of view on the first semantic features and / or the second semantic features based on the feature similarity relationship to obtain aligned features, to achieve the field of view alignment, comprises the following steps: calculating a similarity between the first reference features and the second reference features to obtain a similarity matrix; mapping values of elements in the similarity matrix to values in a preset range to determine the feature similarity relationship.

4. The method of claim 2, wherein, The step of calculating a feature similarity relationship between the first reference features and the second reference features comprises the following steps: acquiring a target row in which a first pixel point in the first reference features of the RGB image is located; extracting a second pixel point of a row corresponding to the target row from the second reference features of the NIR image; calculating a similarity between the first pixel point and the second pixel point to determine the feature similarity relationship.

5. The method of claim 2, wherein, The alignment features include first alignment features corresponding to the first semantic features and second alignment features corresponding to the second semantic features, the image enhancement model includes first and second enhancement reconstruction units that have the same structure but different parameters, and the field-of-view alignment and enhancement reconstruction processing of the RGB image and the NIR image by using the image enhancement model to obtain an image enhancement prediction image includes: performing enhancement reconstruction processing on the aggregated features of the first and second semantic features by using the first enhancement reconstruction unit to obtain a first image enhancement prediction image; performing enhancement reconstruction processing on the aggregated features of the first and second semantic features by using the first enhancement reconstruction unit to obtain a first image enhancement prediction image.

6. The method of claim 5, wherein, The reference images include first and second reference images, the first and second reference images are located in different fields of view of the binocular camera, and the loss calculation based on the image enhancement prediction image and the reference images to update the parameters of the image enhancement model to determine a target image enhancement model includes: performing first loss calculation based on the first reference image and the first image enhancement prediction image to obtain a first loss; performing second loss calculation based on the second reference image and the second image enhancement prediction image to obtain a second loss; updating the parameters of the image enhancement model based on the first and second losses to determine a target image enhancement model.

7. The method according to any one of claims 1 to 6, characterized in that, The loss calculation based on the image enhancement prediction image and the reference images to update the parameters of the image enhancement model to determine a target image enhancement model includes: performing edge loss calculation on the image enhancement prediction image and the reference images to obtain an edge loss; performing color loss calculation on the image enhancement prediction image and the reference images to obtain a color loss; updating the parameters of the image enhancement model based on the edge loss and the color loss to determine a target image enhancement model.

8. An image enhancement method characterized by, It includes: acquiring first and second images in different fields of view collected by a target binocular camera, the first image being an RGB image and the second image being an NIR image; inputting the first and second images into a target image enhancement model to obtain a target image, the target image enhancement model being trained according to the training method of the image enhancement model in any one of claims 1-7.

9. An electronic device, comprising: It includes: a memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the training method of the image enhancement model in any one of claims 1-7 or the image enhancement method in claim 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to execute the training method of the image enhancement model in any one of claims 1-7 or the image enhancement method in claim 8.

Images