Low-light image enhancement method and device, electronic equipment and readable medium

Abstract

The embodiment of the invention discloses a low-light image enhancement method and device, electronic equipment and a readable medium. A specific embodiment of the method comprises the following steps: acquiring a low-light image and a reference image; performing image enhancement processing on the low-light image; inputting the reference image into a reference image feature extraction layer; executing the following enhancement processing: inputting the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into a conditional information generation module; inputting the fused image feature information and the first low-light image feature information into a feature enhancement module; updating the first low-light image feature information; updating the reference image feature information sequence; performing enhancement processing again based on the updated reference image feature information sequence; determining the updated first low-light image feature information as enhanced low-light image feature information; and performing enhancement processing on the low-light image. According to the embodiment, the enhancement effect of the low-light image is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a low-light image enhancement method, device, electronic device, and computer-readable medium. Background Art

[0002] Low-light image enhancement is a technology for enhancing low-light images. At present, when enhancing low-light images, the commonly used method is to directly adjust the brightness and contrast of the low-light images uniformly to achieve the effect of enhancing the low-light images.

[0003] However, when the above method is used to enhance low-light images, the following technical problems often occur:

[0004] Directly adjusting the brightness and contrast of low-light images in a unified manner to achieve the effect of enhancing low-light images. Directly adjusting the brightness and contrast of low-light images in a unified manner is usually a global adjustment, which may change the relative brightness relationship of different color areas in the low-light image, thereby causing color distortion of the low-light image. At the same time, the unified adjustment of the global brightness and contrast of low-light images ignores the detail information in the low-light image, which may cause overexposure and loss of details in high-brightness areas; in low-brightness areas, the low-light image may still appear blurred due to insufficient brightness improvement, resulting in poor enhancement of the low-light image.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure provide a low-light image enhancement method, an apparatus, an electronic device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a low-light image enhancement method, the method comprising: acquiring a low-light image and a first preset number of reference images; inputting the low-light image into a first convolutional layer and a second convolutional layer included in a pre-trained image enhancement model to obtain first low-light image feature information and second low-light image feature information; for each reference image in the first preset number of reference images, based on the reference image, performing the following image enhancement processing on the low-light image: inputting the reference image into a second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image; based on the reference image feature information sequence and the first low-light image feature information, performing the following enhancement processing: inputting the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into the image enhancement model included in the image enhancement model a condition information generation module to obtain fused image feature information; input the fused image feature information and the first low-light image feature information into the feature enhancement module included in the above-mentioned image enhancement model to obtain enhanced low-light image feature information; based on the enhanced low-light image feature information, update the first low-light image feature information; delete the first reference image feature information from the reference image feature information sequence to update the reference image feature information sequence; in response to determining that the updated reference image feature information sequence is not empty, perform enhancement processing again based on the updated reference image feature information sequence and the updated first low-light image feature information; in response to determining that the updated reference image feature information sequence is empty, determine the updated first low-light image feature information as the enhanced low-light image feature information; based on the enhanced low-light image feature information, perform enhancement processing on the low-light image to obtain an enhanced low-light image consistent with the image style of the reference image.

[0009] In a second aspect, some embodiments of the present disclosure provide a low-light image enhancement device, the device comprising: an acquisition unit, configured to acquire a low-light image and a first preset number of reference images; an input unit, configured to input the low-light image into a first convolutional layer and a second convolutional layer included in a pre-trained image enhancement model to obtain first low-light image feature information and second low-light image feature information; an enhancement processing unit, configured to perform the following image enhancement processing on the low-light image based on the reference image for each reference image in the first preset number of reference images: input the reference image into a second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image; based on the reference image feature information sequence and the first low-light image feature information, perform the following enhancement processing: convert the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into a feature extraction layer; The condition information is input into a condition information generation module included in the above-mentioned image enhancement model to obtain fused image feature information; the fused image feature information and the first low-light image feature information are input into a feature enhancement module included in the above-mentioned image enhancement model to obtain enhanced low-light image feature information; based on the enhanced low-light image feature information, the first low-light image feature information is updated; the first reference image feature information is deleted from the reference image feature information sequence to update the reference image feature information sequence; in response to determining that the updated reference image feature information sequence is not empty, enhancement processing is performed again based on the updated reference image feature information sequence and the updated first low-light image feature information; in response to determining that the updated reference image feature information sequence is empty, the updated first low-light image feature information is determined as the enhanced low-light image feature information; based on the enhanced low-light image feature information, the low-light image is enhanced to obtain an enhanced low-light image consistent with the image style of the reference image.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the low-light image enhancement method of some embodiments of the present disclosure, the color distortion of the low-light image is reduced and the enhancement effect of the low-light image is improved. Specifically, the reason for the color distortion of the low-light image and the poor enhancement effect of the low-light image is that the brightness and contrast of the low-light image are directly adjusted uniformly to achieve the effect of enhancing the low-light image. The direct uniform adjustment of the brightness and contrast of the low-light image is usually a global adjustment, which may change the relative brightness relationship of different color areas in the low-light image, thereby causing the color distortion of the low-light image. At the same time, the global brightness and contrast of the low-light image are uniformly adjusted, ignoring the detail information in the low-light image, which may cause overexposure and loss of details in the high-brightness area; in the low-brightness area, the low-light image may still appear blurred due to insufficient brightness improvement, thereby causing the enhancement effect of the low-light image to be poor. Based on this, the low-light image enhancement method of some embodiments of the present disclosure first obtains a low-light image and a first preset number of reference images. Thus, a first preset number of reference images for enhancing the low-light image can be obtained. Then, the low-light image is input into the first convolution layer and the second convolution layer included in the pre-trained image enhancement model to obtain the first low-light image feature information and the second low-light image feature information. Thus, the first low-light image feature information and the second low-light image feature information of the low-light image can be characterized. The first low-light image feature information and the second low-light image feature information can include detail feature information such as edges, textures, corners, etc. of the low-light image. Then, for each reference image in the first preset number of reference images, based on the reference image, the following image enhancement processing is performed on the low-light image: Step 1, input the reference image into the second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image. Thus, the reference image feature information at different scales of the reference image can be extracted through the second preset number of reference image feature extraction layers to obtain a reference image feature information sequence. Based on the reference image feature information sequence and the first low-light image feature information, the following enhancement processing is performed: First, the second low-light image feature information and the first reference image feature information in the reference image feature information sequence are input into the condition information generation module included in the above-mentioned image enhancement model to obtain fused image feature information. Thus, the reference image feature information of the low-light image can be integrated into the second low-light image feature information of the low-light image to obtain fused image feature information for generating enhanced low-light image feature information. Through the condition information fusion module, the color information of the reference image can be combined with the second low-light image feature information of the low-light image, thereby ensuring that the accuracy and naturalness of the image color are maintained while the brightness is subsequently enhanced.Then, the fused image feature information and the first low-light image feature information are input into the feature enhancement module included in the above-mentioned image enhancement model to obtain enhanced low-light image feature information. Thus, enhanced low-light image feature information for updating the first low-light image feature information can be obtained. Then, based on the enhanced low-light image feature information, the first low-light image feature information is updated. Thus, the first low-light image feature information can be updated so that the first low-light image feature information simultaneously includes feature information of the reference image (i.e., at least one reference image feature information) and low-light image detail feature information. Next, the first reference image feature information is deleted from the reference image feature information sequence to update the reference image feature information sequence. Then, in response to determining that the updated reference image feature information sequence is not empty, enhancement processing is performed again based on the updated reference image feature information sequence and the updated first low-light image feature information. Then, in response to determining that the updated reference image feature information sequence is empty, the updated first low-light image feature information is determined as enhanced low-light image feature information. Thus, the enhanced low-light image feature information that includes both the feature information of the reference image (i.e., at least one reference image feature information) and the detail feature information of the low-light image can be obtained through the above-mentioned enhancement processing. Based on the enhanced low-light image feature information, the low-light image is enhanced to obtain an enhanced low-light image that is consistent with the image style of the reference image. Thus, the low-light image can be enhanced by using the enhanced low-light image feature information that includes both the feature information of the reference image (i.e., at least one reference image feature information) and the detail feature information of the low-light image, and the color information of the reference image is combined with the features of the low-light image to enhance the brightness and contrast while maintaining the accuracy and naturalness of the color of the low-light image, thereby reducing the degree of color distortion of the low-light image. At the same time, the detail feature information of the low-light image is taken into account in the process of enhancing the low-light image, thereby reducing the possibility of overexposure in high-brightness areas and insufficient enhancement in low-brightness areas, thereby improving the enhancement effect of the low-light image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of the low-light image enhancement method according to the present disclosure;

[0015] Figure 2 is a schematic structural diagram of some embodiments of the low-light image enhancement device according to the present disclosure;

[0016] Figure 3is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure;

[0017] Figure 4 is a schematic diagram of a model structure according to some embodiments of the image enhancement model disclosed herein;

[0018] Figure 5 is a schematic diagram of the module structure of some embodiments of the condition information generation module included in the image enhancement model of the present disclosure;

[0019] Figure 6 It is a schematic diagram of the module structure of some embodiments of the feature enhancement module included in the image enhancement model according to the present disclosure. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0022] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0025] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0026] Figure 1 A process 100 of some embodiments of the low-light image enhancement method according to the present disclosure is shown. The low-light image enhancement method comprises the following steps:

[0027] Step 101: Acquire a low-light image and a first preset number of reference images.

[0028] In some embodiments, the execution subject (e.g., a computing device) of the low-light image enhancement method may obtain a low-light image and a first preset number of reference images through a wired connection or a wireless connection. The low-light image may be an image taken in a low-light environment (e.g., at night). In practice, the execution subject may obtain the low-light image through a camera device. The execution subject may obtain a first preset number of reference images from a preset database. Each of the first preset number of reference images may be an image taken in a normal environment.

[0029] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0030] Step 102: input the low-light image into a first convolutional layer and a second convolutional layer included in a pre-trained image enhancement model to obtain first low-light image feature information and second low-light image feature information.

[0031] In some embodiments, the execution subject may input the low-light image into the first convolution layer and the second convolution layer included in the pre-trained image enhancement model to obtain the first low-light image feature information and the second low-light image feature information. Figure 4 Schematic diagrams of the model structures of some embodiments of the image enhancement model disclosed in the present invention are shown. Figure 4 As shown. The first convolution layer and the second convolution layer may be convolution layers for extracting features from low-light images. The first low-light image feature information may represent a feature map of the low-light image. The first convolution layer may be the same as the second convolution layer, or the first convolution layer may be different from the second convolution layer. The second low-light image feature information may represent a feature map obtained by extracting features from the low-light image. The image enhancement model may be a neural network model for performing image enhancement on low-light images according to a first preset number of reference images to obtain each enhanced low-light image. The image enhancement model may include a first convolution layer and a second convolution layer, a conditional information generation module, and a feature enhancement module. The output information of the second convolution layer and the reference image feature information are input to the conditional information generation module. The output information of the conditional information generation module and the output information of the first convolution layer may be output to the feature enhancement module. The reference image feature information may be a feature map representing a reference image.

[0032] Step 103: For each reference image in the first preset number of reference images, based on the reference image, perform the following image enhancement processing on the low-light image:

[0033] Step 1031: input the reference image to a second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image.

[0034] In some embodiments, the execution entity may input the reference image into a second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image. The second preset number of reference image feature extraction layers may be convolutional layers for extracting features from the reference image. The reference image feature information sequence may characterize the image features of the reference image at various scales. Each reference image feature information in the reference image feature information sequence may represent features of the reference image extracted at a single scale (such as edges, colors, contrast, etc.).

[0035] Step 1032: Based on the reference image feature information sequence and the first low-light image feature information, perform the following enhancement processing:

[0036] Step 10321: input the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into a condition information generation module included in the image enhancement model to obtain fused image feature information.

[0037] In some embodiments, the execution subject may input the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into the condition information generation module included in the image enhancement model to obtain fused image feature information. Figure 5 FIG. 1 is a schematic diagram of the module structure of some embodiments of the condition information generation module included in the image enhancement model of the present disclosure. Figure 5 As shown. The condition information generation module includes a low-light image feature pooling layer, a reference image feature pooling layer, a feature fusion layer, and a fully connected layer. The low-light image feature pooling layer may be a pooling layer that performs maximum pooling processing on the second low-light image feature information. The reference image feature pooling layer may be a pooling layer that performs maximum pooling processing on the reference image feature information. The feature fusion layer may be a Concat layer that performs a connection operation on the pooled second low-light image feature information and the pooled reference image feature information.

[0038] In some optional implementations of some embodiments, the execution subject may input the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into the condition information generation module included in the image enhancement model to obtain the fused image feature information through the following steps:

[0039] In the first step, the second low-light image feature information is input into the low-light image feature pooling layer to obtain pooled low-light image feature information corresponding to the second low-light image feature information. The pooled low-light image feature information may be information obtained by performing maximum pooling processing on the second low-light image feature information.

[0040] In the second step, the first reference image feature information in the reference image feature information sequence is determined as the target reference image feature information.

[0041] In the third step, the target reference image feature information is input into the reference image feature pooling layer to obtain pooled reference image feature information corresponding to the target reference image feature information. The pooled reference image feature information may be information obtained by performing maximum pooling processing on the target reference image feature information.

[0042] In the fourth step, the pooled low-light image feature information and the pooled reference image feature information are input into the feature fusion layer to obtain fused image feature information. The fused feature information may be a feature map obtained by splicing the pooled low-light image feature information and the pooled reference image feature information. In practice, after the pooled low-light image feature information and the pooled reference image feature information are input into the feature fusion layer, the feature fusion layer may splice the pooled low-light image feature information and the pooled reference image feature information in Num dimension or Channel dimension to obtain fused image feature information. As an example, the pooled low-light image feature information may be (N, C1, H, W), and the pooled reference image feature information may be (N, C2, H, W). The N may be the number of feature maps represented by the low-light image feature information and the pooled reference image feature information. The C1 may be the number of channels of the feature map represented by the pooled low-light image feature information, representing the feature information of the low-light image (such as edge, color, contrast, etc.). The above C2 may be the number of channels represented by the pooled reference image feature information, indicating the feature information of the reference image (such as edge, color, contrast, etc.). The above H and W respectively represent the height and width of the feature map represented by the low-light image feature information and the pooled reference image feature information. The above fused image feature information may be (N, C1+C2, H, W).

[0043] In the fifth step, the fused image feature information is input into the fully connected layer to obtain image enhancement guidance information, wherein the image enhancement guidance information may be the fused image feature information after nonlinear transformation and integration.

[0044] The sixth step is to generate fused image feature information based on the above image enhancement guidance information and the above pooled low-light image feature information. In practice, the above execution entity may perform element-wise addition of the feature map represented by the above image enhancement guidance information and the feature map represented by the above pooled low-light image feature information to obtain fused image feature information. The above fused image feature information may be a feature map obtained by performing element-wise addition of the feature map represented by the above image enhancement guidance information and the feature map represented by the above pooled low-light image feature information.

[0045] In some optional implementations of some embodiments, the execution subject may generate fused image feature information based on the image enhancement guidance information and the pooled low-light image feature information through the following steps:

[0046] The first step is to fuse the image enhancement guidance information with the pooled low-light image feature information to obtain initial fused image feature information. In practice, the execution entity may perform matrix multiplication of the feature map represented by the image enhancement guidance information with the feature map represented by the pooled low-light image feature information to obtain initial fused image feature information.

[0047] In the second step, the initial fused image feature information and the image enhancement guidance information are added element-wise to obtain fused image feature information.

[0048] Step 10322: Input the fused image feature information and the first low-light image feature information into a feature enhancement module included in the image enhancement model to obtain enhanced low-light image feature information.

[0049] In some embodiments, the execution subject may input the fused image feature information and the first low-light image feature information into the feature enhancement module included in the image enhancement model to obtain enhanced low-light image feature information. Figure 6 FIG. 1 is a schematic diagram of the module structure of some embodiments of the feature enhancement module included in the image enhancement model of the present disclosure. Figure 6The feature enhancement module shown includes a first feature enhancement layer 601, a second feature enhancement layer 602 and a third feature enhancement layer 603. The first feature enhancement layer 601 includes the first residual context module of the enhancement layer 1, the second residual context module of the enhancement layer 1 and the conditional aggregation module of the enhancement layer 1. The second residual context module of the enhancement layer 1 receives the first enhanced feature map, and the conditional aggregation module of the enhancement layer 1 receives the information output by the second residual context module of the enhancement layer 1. Each residual context module of the first residual context module of the enhancement layer 1, the second residual context module of the enhancement layer 1, the first residual context module of the enhancement layer 2, the second residual context module of the enhancement layer 2, the first residual context module of the enhancement layer 3 and the second residual context module of the enhancement layer 3 can divide the information input to the residual context module into two branches for operation, wherein one branch sequentially applies convolution, Reshape and tpgunay operations to the information input to the residual context module to obtain the first branch feature information. After the other branch applies the Reshape operation to the information input to the residual context module, the second branch feature information is obtained, the feature information generated by the two branches (the first branch feature information and the second branch feature information) is fused, and the global feature information is obtained by applying matrix multiplication. Then, the global feature information is convolved through two convolutional layers to obtain the global convolution feature information. The above-mentioned residual context module is used to extract the features in the feature enhancement layer and suppress the further forward transmission of useless features. The above-mentioned execution subject can determine the information obtained after applying the convolution operation to the information input to the residual context module as the initial feature information. Finally, the above-mentioned auxiliary subject can perform element-wise addition of the initial feature information and the global convolution feature information to aggregate the global convolution feature information into the initial feature information. The conditional aggregation module in the above-mentioned enhancement layer one conditional aggregation module and the enhancement layer two conditional aggregation module can be used to introduce the features of the reference image into the low-light image. The above-mentioned conditional aggregation module may include a fully connected layer and an activation function layer. The conditional aggregation module can perform fusion processing on the fused image feature information and the received information. Each of the above-mentioned first feature aggregation module of the enhancement layer two, the above-mentioned second feature aggregation module of the enhancement layer two, the first feature aggregation module of the enhancement layer three, and the second feature aggregation module of the enhancement layer three consists of two parts: feature fusion and feature calibration. Among them, the purpose of the feature fusion operation is to combine the information from different feature enhancement layers (resolution streams) to generate new feature identifiers. The feature calibration operation uses these identifiers to readjust the feature maps from different streams. The above-mentioned feature aggregation module can be used to fuse features of different resolution streams and aggregate features of different scales using the self-attention mechanism. The above-mentioned feature aggregation module can be a multi-scale feature fusion module.

[0050] In some optional implementations of some embodiments, the execution subject may input the fused image feature information and the first low-light image feature information into a feature enhancement module included in the image enhancement model to obtain enhanced low-light image feature information through the following steps:

[0051] The first step is to downsample the first low-light image feature information according to the first preset downsampling rate information, the second preset downsampling rate information, and the third preset downsampling rate information to obtain the first feature map, the second feature map, and the third feature map. For example, the first preset downsampling rate information may be "the downsampling rate is 2", the second preset downsampling rate information may be "the downsampling rate is 4", and the third preset downsampling rate information may be "the downsampling rate is 8". In practice, the execution subject may downsample the first low-light image feature information according to the first preset downsampling rate information to obtain the third feature map. Then, the execution subject may downsample the first feature map according to the second preset downsampling rate information to obtain the second feature map. Afterwards, the execution subject may downsample the second feature map to obtain the first feature map.

[0052] The second step is to input the first feature map into the first residual context module of the enhancement layer of the first feature enhancement layer to obtain the first enhanced feature map. The first feature enhancement layer includes the first residual context module of the enhancement layer, the second residual context module of the enhancement layer, and the conditional aggregation module of the enhancement layer. The second residual context module of the enhancement layer receives the first enhanced feature map, the conditional aggregation module of the enhancement layer receives the information output by the second residual context module of the enhancement layer, and the conditional aggregation module of the enhancement layer performs a fusion process on the fused image feature information and the received information.

[0053] The third step is to input the second feature map into the first residual context module of the second feature enhancement layer to obtain the second enhanced feature map. The second feature enhancement layer 602 includes the first residual context module of the second enhancement layer, the first feature aggregation module of the second enhancement layer, the second residual context module of the second enhancement layer, the second feature aggregation module of the second enhancement layer, and the conditional aggregation module of the second enhancement layer. The first feature aggregation module of the second enhancement layer receives the upsampling information of the second enhancement feature map and the first enhancement feature map, and the first feature aggregation module of the second enhancement layer performs aggregation processing on the received information. The second residual context module of the second enhancement layer receives the information output by the first feature aggregation module of the second enhancement layer, the second feature aggregation module of the second enhancement layer receives the upsampling information of the output information of the second residual context module of the second enhancement layer and the output information of the conditional aggregation module of the first enhancement layer, and the second feature aggregation module of the second enhancement layer performs aggregation processing on the received information. The conditional aggregation module of the second enhancement layer receives the information output by the second feature aggregation module of the second enhancement layer, and the conditional aggregation module of the second enhancement layer fuses the feature information of the fused image with the information output by the second feature aggregation module of the second enhancement layer;

[0054] The fourth step is to input the third feature map into the first residual context module of the enhancement layer three of the third feature enhancement layer to obtain the third enhanced feature map. The third feature enhancement layer 603 includes the first residual context module of the enhancement layer three, the first feature aggregation module of the enhancement layer three, the second residual context module of the enhancement layer three, and the second feature aggregation module of the enhancement layer three. The first feature aggregation module of the enhancement layer three receives the output information of the first residual context module of the enhancement layer three and the up-sampled information of the output information of the first feature aggregation module of the enhancement layer two, and the first feature aggregation module of the enhancement layer three aggregates the received information. The second residual context module of the enhancement layer three receives the output information of the first feature aggregation module of the enhancement layer three, the second feature aggregation module of the enhancement layer three receives the up-sampled information of the output information of the second residual context module of the enhancement layer three and the output information of the conditional aggregation module of the enhancement layer two, and aggregates the received information to obtain enhanced low-light image feature information.

[0055] Step 10323: Update the first low-light image feature information based on the enhanced low-light image feature information.

[0056] In some embodiments, the execution entity may update the first low-light image feature information based on the enhanced low-light image feature information.

[0057] In some optional implementations of some embodiments, the execution subject may update the first low-light image feature information based on the enhanced low-light image feature information through the following steps:

[0058] In the first step, the enhanced low-light image feature information is input into the convolutional layer connected to the feature enhancement module to obtain updated low-light image feature information.

[0059] In a second step, the updated low-light image feature information is updated to the first low-light image feature information, so as to update the first low-light image feature information.

[0060] Step 10324: Delete the first reference image feature information from the reference image feature information sequence to update the reference image feature information sequence.

[0061] In some embodiments, the execution entity may delete the first reference image feature information from the reference image feature information sequence to update the reference image feature information sequence.

[0062] Step 10325, in response to determining that the updated reference image feature information sequence is not empty, performing enhancement processing again based on the updated reference image feature information sequence and the updated first low-light image feature information.

[0063] In some embodiments, the execution subject may, in response to determining that the updated reference image feature information sequence is not empty, perform enhancement processing again based on the updated reference image feature information sequence and the updated first low-light image feature information.

[0064] Step 10326 , in response to determining that the updated reference image feature information sequence is empty, determining the updated first low-light image feature information as enhanced low-light image feature information.

[0065] In some embodiments, the execution subject may determine the updated first low-light image feature information as enhanced low-light image feature information in response to determining that the updated reference image feature information sequence is empty.

[0066] Step 1033 , based on the enhanced low-light image feature information, the low-light image is enhanced to obtain an enhanced low-light image having the same image style as the reference image.

[0067] In some embodiments, the above-mentioned execution entity can enhance the low-light image based on the enhanced low-light image feature information to obtain an enhanced low-light image with the same image style as the reference image.

[0068] In some optional implementations of some embodiments, the execution subject may perform enhancement processing on the low-light image based on the enhanced low-light image feature information through the following steps to obtain an enhanced low-light image that is consistent with the image style of the reference image:

[0069] The first step is to generate an enhanced residual image based on the enhanced low-light image feature information. In practice, the execution entity may input the enhanced low-light image feature information into a pre-trained residual network to obtain an enhanced residual image corresponding to the enhanced low-light image feature information. The pre-trained residual network is used to characterize the correspondence between the enhanced low-light image feature information and the enhanced residual image. The enhanced residual image may be a residual map for enhancing the low-light image.

[0070] In the second step, based on the enhanced residual image, the low-light image is enhanced to obtain an enhanced low-light image that is consistent with the image style of the reference image. In practice, the execution subject may add the low-light image and the enhanced residual image pixel by pixel to obtain an enhanced low-light image that is consistent with the image style of the reference image. That is, for each low-light pixel in the low-light image, the execution subject may determine the enhanced residual image pixel in the enhanced residual image that is at the same position as the low-light pixel as the enhanced pixel. Afterwards, the execution subject may determine the sum of the pixel value represented by the pixel value information of the low-light pixel and the pixel value represented by the enhanced pixel information as the updated pixel value information. Then, the execution subject may determine the updated pixel value information as the pixel value information of the low-light pixel to update the low-light pixel. Finally, the execution subject may determine the low-light image after each low-light pixel is updated as an enhanced low-light image that is consistent with the image style of the reference image. As an example, the pixel value information of the above-mentioned low-light pixel can be (18, 12, 5), the pixel value information of the enhanced pixel at the same position as the low-light pixel in the enhanced residual image can be (2, 4, 2), and the updated pixel value information can be (20, 16, 7).

[0071] In the process of adopting technical solutions to solve the problems mentioned in the background technology, the following problems are often accompanied:

[0072] When enhancing the low-light images, enhanced low-light images with the same style as the reference images are obtained, but no quality detection is performed on the enhanced low-light images after enhancement. Some images in the enhanced low-light images after enhancement may still have poor quality problems such as darkness, blur, loss of details, etc., resulting in poor overall image enhancement effect of the low-light images.

[0073] Faced with the above technical problems, we decided to adopt the following solutions:

[0074] In some optional implementations of some embodiments, after performing the following image enhancement processing on the low-light image based on each reference image in the first preset number of reference images, the execution subject may further perform the following steps:

[0075] In the first step, each generated enhanced low-light image is determined as each enhanced low-light image to be detected.

[0076] In the second step, for each of the above enhanced low-light images to be detected, the following detection steps are performed:

[0077] The first sub-step is to segment the enhanced low-light image to be detected to obtain a set of enhanced low-light image blocks to be detected. In practice, the execution subject may segment the enhanced low-light image to be detected to obtain a set of enhanced low-light image blocks to be detected by using image segmentation technology.

[0078] The second sub-step is to perform frequency domain conversion processing on each enhanced low-light image block to be detected in the enhanced low-light image block set to be detected, and obtain frequency domain matrix information corresponding to the enhanced low-light image block to be detected. In practice, the execution subject may perform frequency domain conversion processing on the enhanced low-light image block to be detected by DCT transformation technology to obtain frequency domain matrix information corresponding to the enhanced low-light image block to be detected. The frequency domain matrix information may be a discrete cosine transform coefficient matrix.

[0079] In the third sub-step, for each of the obtained frequency domain matrix information, the frequency domain matrix information and the enhanced low-light image block to be detected corresponding to the frequency domain matrix information are input into a pre-trained image block quality detection model to obtain image block quality detection information. The image block quality detection model may be an image quality assessment (IQA) model that takes the frequency domain matrix information of the image as input and the image block quality detection information as output. The image block quality detection information may be a score for the enhanced low-light image block to be detected.

[0080] The fourth sub-step is to generate image quality score information corresponding to the enhanced low-light image to be detected based on the obtained quality detection information of each image block. In practice, the execution subject may determine the average of each quality score corresponding to the quality detection information of each image block as the image quality score information.

[0081] In the third step, the obtained image quality score information is determined as an image quality score information set.

[0082] In the fourth step, each piece of image quality score information represented by the image quality score information in the above-mentioned image quality score information set and having an image quality score less than or equal to a preset value is determined as each piece of filtered image quality score information.

[0083] In the fifth step, the generated enhanced low-light images are determined as an enhanced low-light image set.

[0084] Step 6: For each of the above-mentioned filtered image quality score information, perform the following processing:

[0085] Sub-step one: determining the enhanced low-light image corresponding to the above-mentioned filtered image quality score information in the above-mentioned enhanced low-light image set as the initial enhanced low-light image.

[0086] Sub-step 2: super-resolution reconstruction is performed on the initial enhanced low-light image to obtain a restored enhanced low-light image. In practice, the execution subject may perform super-resolution reconstruction on the initial enhanced low-light image using an image super-resolution reconstruction algorithm to obtain a restored enhanced low-light image.

[0087] Sub-step three: replacing the initial enhanced low-light image in the enhanced low-light image set with the repaired enhanced low-light image to update the enhanced low-light image set.

[0088] In the seventh step, the updated enhanced low-light image set is determined as the enhanced image set corresponding to the low-light image.

[0089] The above technical solution and its related contents, as an inventive point of the embodiments of the present disclosure, solve the technical problem of "the overall image enhancement effect of low-light images is poor". The factors that lead to the poor overall image enhancement effect of low-light images are often as follows: when enhancing the low-light images, each enhanced low-light image consistent with the style of each reference image is obtained, and the quality of the enhanced low-light images after enhancement is not detected, and some images in each enhanced low-light image after enhancement may still have poor quality problems such as dimness, blur, and loss of details, resulting in poor overall image enhancement effect of low-light images. If the above factors are solved, the overall image enhancement effect of low-light images can be improved. In order to achieve this effect, first, each generated enhanced low-light image is determined as each enhanced low-light image to be detected. Then, for each of the enhanced low-light images to be detected, the following detection steps are performed: In the first step, the enhanced low-light image to be detected is segmented to obtain a set of enhanced low-light image blocks to be detected. Thus, a set of enhanced low-light image blocks to be detected for quality detection of the enhanced low-light image to be detected can be obtained. The second step is to perform frequency domain conversion processing on each enhanced low-light image block to be detected in the above-mentioned enhanced low-light image block set to be detected, and obtain frequency domain matrix information corresponding to the above-mentioned enhanced low-light image block to be detected. The third step is to input the above-mentioned frequency domain matrix information and the enhanced low-light image block to be detected corresponding to the above-mentioned frequency domain matrix information into a pre-trained image block quality detection model for each frequency domain matrix information obtained, and obtain image block quality detection information. Therefore, the frequency domain matrix information of the image block can be input into the image block quality detection model to detect the quality of the image block and obtain image block quality detection information. The third step is to generate image quality score information related to the above-mentioned enhanced low-light image to be detected based on the quality detection information of each image block obtained. Therefore, through the above-mentioned detection steps, the quality of each enhanced low-light image to be detected can be detected, and each image quality score information of each enhanced low-light image to be detected can be obtained. Then, each obtained image quality score information is determined as an image quality score information set. Next, each image quality score information represented by the image quality score information in the above-mentioned image quality score information set and having an image quality score less than or equal to a preset value is determined as each filtered image quality score information. Thus, each filtered image quality score information with a poor image quality score can be determined. Then, each generated enhanced low-light image is determined as an enhanced low-light image set. Thereafter, for each filtered image quality score information in the above-mentioned filtered image quality score information, the following processing is performed: In the first step, the enhanced low-light image corresponding to the above-mentioned filtered image quality score information in the above-mentioned enhanced low-light image set is determined as the initial enhanced low-light image. Thus, the initial enhanced low-light image with poor image quality corresponding to the filtered image quality score information can be determined from the enhanced low-light image set.Then, the above-mentioned initial enhanced low-light image is subjected to super-resolution reconstruction processing to obtain a repaired enhanced low-light image. Thus, the initial enhanced low-light image with poor image quality can be subjected to super-resolution reconstruction processing to improve the image quality of the initial enhanced low-light image. Next, the initial enhanced low-light image in the above-mentioned enhanced low-light image set is replaced with the repaired enhanced low-light image to update the enhanced low-light image set. Then, the updated enhanced low-light image set is determined as the enhanced image set corresponding to the above-mentioned low-light image. Also, after the low-light image is subjected to image enhancement processing to obtain each enhanced low-light image, the image quality of each enhanced low-light image is detected to obtain each image quality score information. And according to each image quality score information, the enhanced low-light image with poor image quality in each enhanced low-light image is subjected to super-resolution reconstruction processing to obtain an enhanced image set corresponding to the low-light image. The image quality of the enhanced low-light image with poor image quality in each enhanced low-light image is improved, thereby improving the overall image enhancement effect of the low-light image.

[0090] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the low-light image enhancement method of some embodiments of the present disclosure, the color distortion of the low-light image is reduced and the enhancement effect of the low-light image is improved. Specifically, the reason for the color distortion of the low-light image and the poor enhancement effect of the low-light image is that the brightness and contrast of the low-light image are directly adjusted uniformly to achieve the effect of enhancing the low-light image. The direct uniform adjustment of the brightness and contrast of the low-light image is usually a global adjustment, which may change the relative brightness relationship of different color areas in the low-light image, thereby causing the color distortion of the low-light image. At the same time, the global brightness and contrast of the low-light image are uniformly adjusted, ignoring the detail information in the low-light image, which may cause overexposure and loss of details in the high-brightness area; in the low-brightness area, the low-light image may still appear blurred due to insufficient brightness improvement, thereby causing the enhancement effect of the low-light image to be poor. Based on this, the low-light image enhancement method of some embodiments of the present disclosure first obtains a low-light image and a first preset number of reference images. Thus, a first preset number of reference images for enhancing the low-light image can be obtained. Then, the low-light image is input into the first convolution layer and the second convolution layer included in the pre-trained image enhancement model to obtain the first low-light image feature information and the second low-light image feature information. Thus, the first low-light image feature information and the second low-light image feature information of the low-light image can be characterized. The first low-light image feature information and the second low-light image feature information can include detail feature information such as edges, textures, corners, etc. of the low-light image. Then, for each reference image in the first preset number of reference images, based on the reference image, the following image enhancement processing is performed on the low-light image: Step 1, input the reference image into the second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image. Thus, the reference image feature information at different scales of the reference image can be extracted through the second preset number of reference image feature extraction layers to obtain a reference image feature information sequence. Based on the reference image feature information sequence and the first low-light image feature information, the following enhancement processing is performed: First, the second low-light image feature information and the first reference image feature information in the reference image feature information sequence are input into the condition information generation module included in the above-mentioned image enhancement model to obtain fused image feature information. Thus, the reference image feature information of the low-light image can be integrated into the second low-light image feature information of the low-light image to obtain fused image feature information for generating enhanced low-light image feature information. Through the condition information fusion module, the color information of the reference image can be combined with the second low-light image feature information of the low-light image, thereby ensuring that the accuracy and naturalness of the image color are maintained while the brightness is subsequently enhanced.Then, the fused image feature information and the first low-light image feature information are input into the feature enhancement module included in the above-mentioned image enhancement model to obtain the enhanced low-light image feature information. Thus, the enhanced low-light image feature information for updating the first low-light image feature information can be obtained. Then, based on the enhanced low-light image feature information, the first low-light image feature information is updated. Thus, the first low-light image feature information can be updated so that the first low-light image feature information simultaneously includes the feature information of the reference image (i.e., at least one reference image feature information) and the low-light image detail feature information. Next, the first reference image feature information is deleted from the reference image feature information sequence to update the reference image feature information sequence. Then, in response to determining that the updated reference image feature information sequence is not empty, the enhancement process is performed again based on the updated reference image feature information sequence and the updated first low-light image feature information. Then, in response to determining that the updated reference image feature information sequence is empty, the updated first low-light image feature information is determined as the enhanced low-light image feature information. Thus, the enhanced low-light image feature information that includes both the feature information of the reference image (i.e., at least one reference image feature information) and the detail feature information of the low-light image can be obtained through the above-mentioned enhancement processing. Based on the enhanced low-light image feature information, the low-light image is enhanced to obtain an enhanced low-light image that is consistent with the image style of the reference image. Thus, the low-light image can be enhanced by using the enhanced low-light image feature information that includes both the feature information of the reference image (i.e., at least one reference image feature information) and the detail feature information of the low-light image, and the color information of the reference image is combined with the features of the low-light image to enhance the brightness and contrast while maintaining the accuracy and naturalness of the color of the low-light image, thereby reducing the degree of color distortion of the low-light image. At the same time, the detail feature information of the low-light image is taken into account in the process of enhancing the low-light image, thereby reducing the possibility of overexposure in high-brightness areas and insufficient enhancement in low-brightness areas, thereby improving the enhancement effect of the low-light image.

[0091] Further references Figure 2 As an implementation of the methods shown in the figures, the present disclosure provides some embodiments of a low-light image enhancement device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0092] like Figure 2As shown, the low-light image enhancement device 200 of some embodiments includes: an acquisition unit 201, an input unit 202 and an enhancement processing unit 203. Among them, the acquisition unit 201 is configured to acquire a low-light image and a first preset number of reference images; the input unit 202 is configured to input the low-light image into a first convolution layer and a second convolution layer included in a pre-trained image enhancement model to obtain first low-light image feature information and second low-light image feature information; the enhancement processing unit 203 is configured to perform the following image enhancement processing on the low-light image based on the reference image for each reference image in the first preset number of reference images: input the reference image into a second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image; based on the reference image feature information sequence and the first low-light image feature information, perform the following enhancement processing: input the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into the first reference image feature information sequence included in the image enhancement model The conditional information generation module included in the image enhancement model is used to obtain fused image feature information; the fused image feature information and the first low-light image feature information are input into the feature enhancement module included in the above-mentioned image enhancement model to obtain enhanced low-light image feature information; based on the enhanced low-light image feature information, the first low-light image feature information is updated; the first reference image feature information is deleted from the reference image feature information sequence to update the reference image feature information sequence; in response to determining that the updated reference image feature information sequence is not empty, the enhancement processing is performed again based on the updated reference image feature information sequence and the updated first low-light image feature information; in response to determining that the updated reference image feature information sequence is empty, the updated first low-light image feature information is determined as the enhanced low-light image feature information; based on the enhanced low-light image feature information, the low-light image is enhanced to obtain an enhanced low-light image consistent with the image style of the reference image.

[0093] It is understood that the units described in the device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 and the units contained therein, and will not be described in detail here.

[0094] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0095] like Figure 3As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0096] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0097] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the functions defined in the method of some embodiments of the present disclosure are executed.

[0098] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0099] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0100] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a low-light image and a first preset number of reference images; inputs the low-light image into the first convolution layer and the second convolution layer included in the pre-trained image enhancement model to obtain first low-light image feature information and second low-light image feature information; for each reference image in the first preset number of reference images, based on the reference image, performs the following image enhancement processing on the low-light image: inputs the reference image into the second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image; based on the reference image feature information sequence and the first low-light image feature information, performs the following enhancement processing: inputs the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into the image enhancement model; The condition information generation module included in the strong model obtains fused image feature information; the fused image feature information and the first low-light image feature information are input into the feature enhancement module included in the above-mentioned image enhancement model to obtain enhanced low-light image feature information; based on the enhanced low-light image feature information, the first low-light image feature information is updated; the first reference image feature information is deleted from the reference image feature information sequence to update the reference image feature information sequence; in response to determining that the updated reference image feature information sequence is not empty, based on the updated reference image feature information sequence and the updated first low-light image feature information, enhancement processing is performed again; in response to determining that the updated reference image feature information sequence is empty, the updated first low-light image feature information is determined as the enhanced low-light image feature information; based on the enhanced low-light image feature information, the low-light image is enhanced to obtain an enhanced low-light image consistent with the image style of the reference image.

[0101] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor includes an acquisition unit, an input unit, and an enhancement processing unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring a low-light image and a first preset number of reference images".

[0104] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0105] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of technical features, but should also cover other technical solutions formed by any combination of technical features or their equivalent features without departing from the inventive concept. For example, a technical solution formed by replacing the features with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A low-light image enhancement method, comprising: Acquire a low-light image and a first preset number of reference images; Inputting the low-light image into a first convolutional layer and a second convolutional layer included in a pre-trained image enhancement model to obtain first low-light image feature information and second low-light image feature information; For each reference image in the first preset number of reference images, based on the reference image, performing the following image enhancement processing on the low-light image: Inputting the reference image into a second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image; Based on the reference image feature information sequence and the first low-light image feature information, the following enhancement processing is performed: Inputting the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into a condition information generation module included in the image enhancement model to obtain fused image feature information; Inputting the fused image feature information and the first low-light image feature information into a feature enhancement module included in the image enhancement model to obtain enhanced low-light image feature information; Based on the enhanced low-light image feature information, updating the first low-light image feature information; Deleting the first reference image feature information from the reference image feature information sequence to update the reference image feature information sequence; In response to determining that the updated reference image feature information sequence is not empty, performing enhancement processing again based on the updated reference image feature information sequence and the updated first low-light image feature information; In response to determining that the updated reference image feature information sequence is empty, determining the updated first low-light image feature information as enhanced low-light image feature information; Based on the enhanced low-light image feature information, the low-light image is enhanced to obtain an enhanced low-light image that is consistent with the image style of the reference image.

2. The method according to claim 1, wherein: The feature enhancement module is connected to the convolution layer, and the updating of the first low-light image feature information based on the enhanced low-light image feature information includes: Inputting the enhanced low-light image feature information into a convolutional layer connected to the feature enhancement module to obtain updated low-light image feature information; The updated low-light image feature information is updated to the first low-light image feature information, so as to update the first low-light image feature information.

3. The method according to claim 1, wherein: The condition information generation module includes a low-light image feature pooling layer, a reference image feature pooling layer, a feature fusion layer, a fully connected layer, and the condition information generation module includes inputting the second low-light image feature information and the first reference image feature information in the reference image feature information sequence into the image enhancement model to obtain fused image feature information, including: Inputting the second low-light image feature information into the low-light image feature pooling layer to obtain pooled low-light image feature information corresponding to the second low-light image feature information; Determining the first reference image feature information in the reference image feature information sequence as the target reference image feature information; Inputting the target reference image feature information into the reference image feature pooling layer to obtain pooled reference image feature information corresponding to the target reference image feature information; Inputting the pooled low-light image feature information and the pooled reference image feature information into the feature fusion layer to obtain fused image feature information; Inputting the fused image feature information into a fully connected layer to obtain image enhancement guidance information; Based on the image enhancement guidance information and the pooled low-light image feature information, fused image feature information is generated.

4. The method according to claim 3, wherein: The generating fused image feature information based on the image enhancement guidance information and the pooled low-light image feature information includes: Fusing the image enhancement guidance information with the pooled low-light image feature information to obtain initial fused image feature information; The initial fused image feature information and the image enhancement guidance information are subjected to element-level addition processing to obtain fused image feature information.

5. The method according to claim 1, wherein: The method of performing enhancement processing on the low-light image based on the enhanced low-light image feature information to obtain an enhanced low-light image consistent with the image style of the reference image includes: Based on the enhanced low-light image feature information, generating an enhanced residual image; Based on the enhanced residual image, the low-light image is enhanced to obtain an enhanced low-light image having a consistent image style with the reference image.

6. The method according to claim 1, wherein: The feature enhancement module includes a first feature enhancement layer, a second feature enhancement layer, and a third feature enhancement layer, and the feature enhancement module includes inputting the fused image feature information and the first low-light image feature information into the image enhancement model to obtain enhanced low-light image feature information, including: Downsampling the first low-light image feature information according to the first preset downsampling rate information, the second preset downsampling rate information, and the third preset downsampling rate information to obtain a first feature map, a second feature map, and a third feature map; Inputting the first feature map to an enhancement layer-first residual context module of the first feature enhancement layer to obtain a first enhanced feature map, wherein the first feature enhancement layer includes the enhancement layer-first residual context module, the enhancement layer-second residual context module, and the enhancement layer-first conditional aggregation module, the enhancement layer-first second residual context module receives the first enhanced feature map, the enhancement layer-first conditional aggregation module receives information output by the enhancement layer-first second residual context module, and the enhancement layer-first conditional aggregation module performs a fusion process on the fused image feature information and the received information; Inputting the second feature map to the first residual context module of the second feature enhancement layer to obtain the second enhanced feature map, wherein the second feature enhancement layer includes the first residual context module of the second enhancement layer, the first feature aggregation module of the second enhancement layer, the second residual context module of the second enhancement layer, the second feature aggregation module of the second enhancement layer, and the conditional aggregation module of the second enhancement layer, the first feature aggregation module of the second enhancement layer receives upsampling information of the second enhancement feature map and the first enhancement feature map, and the first feature aggregation module of the second enhancement layer performs aggregation processing on the received information, the second residual context module of the second enhancement layer receives information output by the first feature aggregation module of the second enhancement layer, the second feature aggregation module of the second enhancement layer receives upsampling information of the output information of the second residual context module of the second enhancement layer and the output information of the conditional aggregation module of the first enhancement layer, and the second feature aggregation module of the second enhancement layer performs aggregation processing on the received information, the conditional aggregation module of the second enhancement layer receives information output by the second feature aggregation module of the second enhancement layer, and the conditional aggregation module of the second enhancement layer fuses the fused image feature information with the received information output by the second feature aggregation module of the second enhancement layer; The third feature map is input into the first residual context module of the enhancement layer three of the third feature enhancement layer to obtain a third enhanced feature map, wherein the third feature enhancement layer includes the first residual context module of the enhancement layer three, the first feature aggregation module of the enhancement layer three, the second residual context module of the enhancement layer three, and the second feature aggregation module of the enhancement layer three, the first feature aggregation module of the enhancement layer three receives the output information of the first residual context module of the enhancement layer three and the upsampling information of the output information of the first feature aggregation module of the enhancement layer two, and the first feature aggregation module of the enhancement layer three performs aggregation processing on the received information, the second residual context module of the enhancement layer three receives the output information of the first feature aggregation module of the enhancement layer three, the second feature aggregation module of the enhancement layer three receives the output information of the second residual context module of the enhancement layer three and the upsampling information of the output information of the conditional aggregation module of the enhancement layer two, and aggregates the received information to obtain enhanced low-light image feature information.

7. A low-light image enhancement device, comprising: an acquisition unit, configured to acquire a low-light image and a first preset number of reference images; An input unit, configured to input the low-light image into a first convolutional layer and a second convolutional layer included in a pre-trained image enhancement model to obtain first low-light image feature information and second low-light image feature information; an enhancement processing unit, configured to perform the following image enhancement processing on the low-light image based on each reference image in the first preset number of reference images: input the reference image into a second preset number of reference image feature extraction layers included in the image enhancement model to obtain a reference image feature information sequence corresponding to the reference image; Based on the reference image feature information sequence and the first low-light image feature information, the following enhancement processing is performed: the second low-light image feature information and the first reference image feature information in the reference image feature information sequence are input into a condition information generation module included in the image enhancement model to obtain fused image feature information; the fused image feature information and the first low-light image feature information are input into a feature enhancement module included in the image enhancement model to obtain enhanced low-light image feature information; based on the enhanced low-light image feature information, the first low-light image feature information is updated; the first reference image feature information is deleted from the reference image feature information sequence to update the reference image feature information sequence; In response to determining that the updated reference image feature information sequence is not empty, performing enhancement processing again based on the updated reference image feature information sequence and the updated first low-light image feature information; In response to determining that the updated reference image feature information sequence is empty, determining the updated first low-light image feature information as enhanced low-light image feature information; Based on the enhanced low-light image feature information, the low-light image is enhanced to obtain an enhanced low-light image that is consistent with the image style of the reference image.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images