Low-illumination image processing method, device, equipment and storage medium

Abstract

The application is suitable for the field of image processing, and provides a low-illumination image processing method, device and equipment and a storage medium. The method comprises the following steps: when a to-be-processed image is received, a plurality of initial pixels corresponding to the to-be-processed image are determined to obtain an initial pixel set; a cascaded relationship of a plurality of low-illumination enhancement mapping tables is determined according to a pre-trained low-illumination enhancement model, wherein the low-illumination enhancement mapping table records the corresponding relationship of pixel values; the plurality of low-illumination enhancement mapping tables are traversed according to the cascaded relationship to obtain a target pixel set corresponding to the initial pixel set; and a target enhanced image corresponding to the to-be-processed image is generated according to the target pixel set. The above method can improve the efficiency of low-illumination enhancement.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a low-illumination image processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Electronic devices can implement application of images in practical scenarios such as autonomous driving, video monitoring, and virtual augmented reality by performing operations such as object detection, classification, identification, scene understanding, and 3D reconstruction on captured images. Image quality has a significant impact on subsequent scene applications, and light changes in a scene often directly affect the quality of captured images. In typical cases, images captured in an insufficient light environment will usually have severe degradation, such as poor visibility, low contrast, and unexpected noise. Weak light undoubtedly reduces the performance of most vision-based algorithms, and therefore, it is necessary to perform low-illumination enhancement processing on images to improve image quality.

[0003] However, low-illumination enhancement processing of images is affected by many factors, and the efficiency of low-illumination enhancement processing cannot be guaranteed.

[0004] SUMMARY

[0005] Embodiments of the present application provide a low-illumination image processing method, a low-illumination image processing device, an electronic device, and a storage medium to solve the problem of poor efficiency of low-illumination enhancement processing.

[0006] The first aspect of the embodiments of the present application provides a low-illumination image processing method, which includes: when a to-be-processed image is received, determining a plurality of initial pixels corresponding to the to-be-processed image to obtain an initial pixel set; determining a cascaded relationship of a plurality of low-illumination enhancement mapping tables according to a pre-trained low-illumination enhancement model, wherein the low-illumination enhancement mapping table records a corresponding relationship of pixel values; traversing the plurality of low-illumination enhancement mapping tables according to the cascaded relationship to obtain a target pixel set corresponding to the initial pixel set; and generating a target enhancement image corresponding to the to-be-processed image according to the target pixel set.

[0007] In the embodiments of the present application, the deep learning method is combined with the mapping table technology to determine the low-illumination enhancement mapping table corresponding to the low-illumination model, and then when the to-be-processed image is received, the low-illumination enhancement mapping table is traversed to obtain the target enhancement image corresponding to the to-be-processed image, which can avoid the problem of complex calculation caused by the deep learning method for low-illumination enhancement of the image, and improve the efficiency of low-illumination enhancement.

[0008] Further, in the method provided by the embodiment of the present application, the pre-trained low-illumination enhancement model comprises: obtaining an initial neural network model; determining an initial convolution kernel structure corresponding to the initial neural network model; adjusting the initial convolution kernel structure to a target convolution kernel structure to obtain a target neural network model; training the target neural network model by taking a historical low-illumination image as input data and a historical low-illumination enhanced image as output data to obtain a low-illumination enhancement model.

[0009] Further, in the method provided by the embodiment of the present application, the initial convolution kernel structure comprises a plurality of 3*3 receptive field convolution kernels, and the adjusting the initial convolution kernel structure to a target convolution kernel structure to obtain a target neural network model comprises: determining a convolution kernel combination of a single 2*2 receptive field convolution kernel and a plurality of 1*1 receptive field convolution kernels; combining the convolution kernel combination according to a first preset number to obtain an intermediate convolution kernel structure; stacking the intermediate convolution kernel structure according to a second preset number to obtain a target convolution kernel structure; and adjusting the 3*3 receptive field convolution kernel in the initial neural network model to the target convolution kernel structure to obtain the target neural network model.

[0010] Further, in the method provided by the embodiment of the present application, before the determining the cascade relationship of a plurality of low-illumination enhancement mapping tables according to the pre-trained low-illumination enhancement model, the method further comprises: determining a convolution kernel calling relationship corresponding to a plurality of intermediate convolution kernel structures in the target convolution kernel structure; calling the intermediate convolution kernel structures to process the historical low-illumination image according to the convolution kernel calling relationship to obtain a plurality of historical low-illumination feature maps; determining a mapping relationship of pixel values between the historical low-illumination image and the historical low-illumination feature maps and between the historical low-illumination feature maps, and constructing the plurality of low-illumination enhancement mapping tables according to the mapping relationship; and combining the low-illumination enhancement mapping tables to obtain a low-illumination enhancement mapping table set.

[0011] Further, in the method provided by the embodiment of the present application, the determining the cascade relationship of a plurality of low-illumination enhancement mapping tables according to the pre-trained low-illumination enhancement model comprises: determining an intermediate convolution kernel structure corresponding to each low-illumination enhancement mapping table; and determining the cascade relationship of the plurality of low-illumination enhancement mapping tables according to a convolution kernel calling relationship of the intermediate convolution kernel structure.

[0012] Further, in the method provided by the embodiment of the present application, the step of traversing the plurality of low-illumination enhancement mapping tables according to the cascade relationship to obtain a target pixel set corresponding to the initial pixel set comprises: determining a first-stage low-illumination enhancement mapping table and a corresponding next-stage low-illumination enhancement mapping table from the plurality of low-illumination enhancement mapping tables according to the cascade relationship; obtaining an initial pixel value corresponding to each initial pixel in the initial pixel set, and traversing the first-stage low-illumination enhancement mapping table according to each initial pixel value in the initial pixel set to obtain a first pixel value; taking the first pixel value obtained by the first-stage traversal as an input of the next-stage traversal, and continuing to traverse the next-stage low-illumination enhancement mapping table according to the first pixel value until a target pixel value corresponding to each initial pixel value is obtained; and combining the target pixel values to obtain a target pixel set.

[0013] Further, in the method provided by the embodiment of the present application, the step of training the target neural network model by taking a historical low-illumination image as input data and a historical low-illumination enhancement image as output data to obtain a low-illumination enhancement model comprises: processing the historical low-illumination image according to a preset self-integration algorithm to obtain an integrated historical low-illumination image; inputting the integrated historical low-illumination image into the target neural network model to obtain a model enhancement image; calculating a loss value according to the historical low-illumination enhancement image and the model enhancement image; and when the loss value is less than a preset loss threshold, taking the target neural network model as the pre-trained low-illumination enhancement model.

[0014] The second aspect of the embodiment of the present application further provides a low-illumination image processing device, which comprises: an image receiving module configured to determine a plurality of initial pixels corresponding to a to-be-processed image to obtain an initial pixel set when the to-be-processed image is received; a mapping table determining module configured to determine a cascade relationship of a plurality of low-illumination enhancement mapping tables according to a pre-trained low-illumination enhancement model, wherein the low-illumination enhancement mapping table records a corresponding relationship of pixel values; a mapping table traversing module configured to traverse the plurality of low-illumination enhancement mapping tables according to the cascade relationship to obtain a target pixel set corresponding to the initial pixel set; and an image enhancement module configured to generate a target enhancement image corresponding to the to-be-processed image according to the target pixel set.

[0015] The third aspect of the embodiment of the present application further provides an electronic device comprising a processor configured to implement the low-illumination image processing method according to any one of the preceding embodiments when executing a computer program stored in a memory.

[0016] The fourth aspect of the embodiment of the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is configured to implement the low-illumination image processing method according to any one of the preceding embodiments when executed by a controller. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an application scenario diagram of the low-light image processing method provided in the embodiments of this application;

[0019] Figure 2 This is a schematic flowchart of the low-light image processing method provided in the embodiments of this application;

[0020] Figure 3 This is a flowchart illustrating the determination of the low-light enhancement model provided in the embodiments of this application;

[0021] Figure 4 This is a flowchart illustrating the determination of the target convolutional kernel structure provided in an embodiment of this application;

[0022] Figure 5A This is a schematic diagram of the convolution kernel combination provided in the embodiments of this application;

[0023] Figure 5B This is a schematic diagram of the intermediate convolutional kernel structure provided in the embodiments of this application;

[0024] Figure 5C This is a schematic diagram of the target convolutional kernel structure provided in the embodiments of this application;

[0025] Figure 6 This is a flowchart illustrating the training process of the low-light enhancement model provided in this application embodiment;

[0026] Figure 7 This is a schematic diagram illustrating the rotation processing of any historical low-light image provided in an embodiment of this application;

[0027] Figure 8 This is the process for determining the low-light enhancement mapping table set provided in the embodiments of this application;

[0028] Figure 9 This is a schematic diagram of the model processing provided in the embodiments of this application;

[0029] Figure 10 This is a flowchart illustrating the determination of the target pixel set provided in an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of the low-light image processing device provided in the embodiments of this application.

[0031] Figure 12 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0033] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application. The described embodiments are part of the present application, but not all embodiments.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments of the present application, and are not intended to limit the present application.

[0035] The existing low-illumination enhancement processing of images is affected by many factors, and the efficiency of low-illumination enhancement processing cannot be guaranteed. Exemplarily, in the related art, the low-illumination enhancement processing of images adopts a deep learning manner, and a general brightness mapping relationship is obtained from big data by deep learning, so as to realize color change to adapt to human visual perception without relying on human participation. However, in some visual tasks, for example, automatic driving, video monitoring and virtual augmented reality scenes all depend on the enhancement result of dark images, and there is a strict limitation on the time resource consumption in the image enhancement process. The huge neural network calculation amount and spatial complexity will result in poor efficiency of enhancement processing, which cannot meet the time resource consumption requirement of the above-mentioned scenes.

[0036] Based on the above problems, an embodiment of the present application provides a low-illumination image processing method to improve the efficiency of low-illumination enhancement processing.

[0037] In combination with Figure 1 The application scenario diagram of the low-illumination image processing method executed by the embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the left image is a to-be-processed image, and the right image is a target enhanced image after enhancement processing. There are a plurality of low-illumination enhancement mapping tables, Figure 1Taking the number of low-illumination enhancement mapping tables as 3 as an example, the three low-illumination enhancement mapping tables are a first low-illumination enhancement mapping table, a second low-illumination enhancement mapping table, and a third low-illumination enhancement mapping table. The first low-illumination enhancement mapping table, the second low-illumination enhancement mapping table, and the third low-illumination enhancement mapping table have a cascading relationship. Each initial pixel value of the to-be-processed image is taken as an index of the first low-illumination enhancement mapping table to obtain a first pixel value. The first pixel value is taken as an index of the second low-illumination enhancement mapping table to obtain a second pixel value. The second pixel value is taken as an index of the third low-illumination enhancement mapping table to obtain a target pixel value. A plurality of target pixel values form a target enhanced image after enhancement processing.

[0038] Figure 2 is a flowchart of a low-illumination image processing method provided by an embodiment of the present application. As shown in Figure 2 , the low-illumination image processing method is executed by an electronic device and can specifically include the following steps. The order of the steps in the flowchart can be changed according to different requirements, and some steps can be omitted.

[0039] S11, when receiving a to-be-processed image, determining a plurality of initial pixels corresponding to the to-be-processed image to obtain an initial pixel set.

[0040] In at least one embodiment of the present application, the to-be-processed image refers to an image that needs to be processed by low-illumination enhancement. The to-be-processed image can be a single image or a video frame image in a video. The image format of the to-be-processed image can be an RGB data format or a YUV data format. The to-be-processed image in the RGB data format contains an R channel, a G channel, and a B channel, which correspond to red, green, and blue three components, respectively, and the color of the image is determined by the values of the three components. The to-be-processed image in the YUV data format includes a Y channel, a U channel, and a V channel, wherein the Y channel represents the brightness of the image, and the U channel and the V channel represent the chroma of the image.

[0041] In an embodiment, the to-be-processed image contains a plurality of initial pixels, and the plurality of initial pixels form an initial pixel set. Each initial pixel corresponds to a brightness value (also referred to as a pixel value in the present application) of the pixel. Illustratively, the brightness value of the pixel of the to-be-processed image is set in the interval [0, 255], wherein the closer to 255, the higher the brightness of the pixel, and the closer to 0, the lower the brightness of the pixel.

[0042] S12, determining a cascading relationship of a plurality of low-illumination enhancement mapping tables according to a pre-trained low-illumination enhancement model, wherein the low-illumination enhancement mapping table records the corresponding relationship of the pixel value.

[0043] In at least one embodiment of the present application, the low-illumination enhancement model refers to a mathematical model used for low-illumination enhancement transformation processing of each pixel in the to-be-processed image. Each initial pixel of the to-be-processed image is processed by calling the low-illumination enhancement model, and a target pixel after enhancement transformation processing can be obtained.

[0044] In an embodiment, the low-illumination enhancement model is a model improved based on the initial neural network model. Considering that the pixel mapping relationship of the image is directly related to the receptive field range, when the pixel receptive field range of the output feature map in deep learning is 9x9 area, an 8-bit size mapping table needs to be used, and the occupied space of the mapping table is extremely large. The present application improves the initial neural network model to obtain the low-illumination enhancement model, which can greatly reduce the occupied space of the mapping table while ensuring the receptive field range, so that the mapping table is more concise, the traversal rate is improved, and thus the low-illumination enhancement processing rate is improved.

[0045] In an embodiment, the initial convolution kernel structure corresponding to the initial neural network model is improved and adjusted to a target convolution kernel structure to obtain a target neural network model. Then, the target neural network model is trained by taking the historical low-illumination image as input data and the historical low-illumination enhancement image as output data, and finally the low-illumination enhancement model is obtained.

[0046] The determination process of the low-illumination enhancement model can refer to the detailed description of the determination process of the low-illumination enhancement model in the following Figure 3 , Figure 4 and Figure 6 .

[0047] In an embodiment, the low-illumination enhancement model includes a target convolution kernel structure, and the target convolution kernel structure includes a plurality of intermediate convolution kernel structures. For each intermediate convolution kernel structure, there is a corresponding low-illumination enhancement mapping table. For example, when the number of intermediate convolution kernel structures is 3, the number of low-illumination enhancement mapping tables is also 3.

[0048] In an embodiment, the low-illumination enhancement mapping table records the correspondence of pixel values, and the low-illumination enhancement mapping table can include the pixel value mapping relationship between the historical low-illumination image and the historical low-illumination feature map, or the pixel value mapping relationship between the historical low-illumination feature maps. For example, when the number of intermediate convolution kernel structures is 3, which are respectively a first intermediate convolution kernel structure, a second intermediate convolution kernel structure and a third intermediate convolution kernel structure. When the historical low-illumination image is input into the low-illumination enhancement model, the first intermediate convolution kernel structure is used to process the historical low-illumination image to obtain a first historical low-illumination feature map; then the second intermediate convolution kernel structure is used to process the first historical low-illumination feature map to obtain a second historical low-illumination feature map; and the second historical low-illumination feature map is input into the third intermediate convolution kernel structure to obtain a third historical low-illumination feature map (i.e. a historical low-illumination enhancement image). At this time, the low-illumination enhancement mapping table set includes a first low-illumination enhancement mapping table, a second low-illumination enhancement mapping table and a third low-illumination enhancement mapping table. The first low-illumination enhancement mapping table includes the mapping relationship between the pixel values of the historical low-illumination image and the first historical low-illumination feature map; the second low-illumination enhancement mapping table includes the mapping relationship between the pixel values of the first historical low-illumination feature map and the second historical low-illumination feature map; and the third low-illumination enhancement mapping table includes the mapping relationship between the pixel values of the second historical low-illumination feature map and the third historical low-illumination feature map.

[0049] In an embodiment, the cascade relationship means that the index of a subsequent mapping table is a value in a previous mapping table. For each intermediate convolution kernel structure, there is a corresponding low-illumination enhancement mapping table. The correspondence can be determined by setting the same code, which can be a number, a letter or a color, without limitation. By querying the same code, the corresponding intermediate convolution kernel structure of each low-illumination enhancement mapping table can be obtained. The cascade relationship of the plurality of low-illumination enhancement mapping tables can be determined according to the convolution kernel calling relationship of the plurality of intermediate convolution kernel structures.

[0050] For example, when the number of intermediate convolution kernel structures is 3, which are respectively a first intermediate convolution kernel structure, a second intermediate convolution kernel structure and a third intermediate convolution kernel structure. The calling relationship is that the second intermediate convolution kernel structure calls the feature map processed by the first intermediate convolution kernel structure, and the third intermediate convolution kernel structure calls the feature map processed by the second intermediate convolution kernel structure. According to the convolution kernel calling relationship, the value in the first low-illumination enhancement mapping table is the index of the second low-illumination enhancement mapping table, and the value in the second low-illumination enhancement mapping table is the index of the third low-illumination enhancement mapping table.

[0051] The determination process of the low-illumination enhancement mapping table set can refer to the detailed description of the determination process of the low-illumination enhancement mapping table set shown in the following process. Figure 8

[0052] ​S13, according to the cascade relationship, traversing the plurality of low-illumination enhancement mapping tables to obtain a target pixel set corresponding to the initial pixel set.

[0053] In at least one embodiment of the present application, according to the cascade relationship, a first-stage low-illumination enhancement mapping table and a corresponding next-stage low-illumination enhancement mapping table are determined from the plurality of low-illumination enhancement mapping tables; an initial pixel value corresponding to each initial pixel in the initial pixel set is obtained, and a first pixel value is obtained by traversing the first-stage low-illumination enhancement mapping table according to each initial pixel value in the initial pixel set; the first pixel value obtained by traversing the first stage is taken as an input of the next-stage traversal, and the next-stage low-illumination enhancement mapping table is traversed according to the first pixel value until a target pixel value corresponding to each initial pixel value is obtained; and the target pixel values are combined to obtain a target pixel set.

[0054] S14, generating a target enhanced image corresponding to the to-be-processed image according to the target pixel set.

[0055] In at least one embodiment of the present application, the target pixel value in the target pixel set is obtained after each initial pixel in the to-be-processed image is enhanced, and for each initial pixel value in the to-be-processed image, there is a corresponding target pixel value. It can be understood that the target enhanced image corresponding to the to-be-processed image after the enhancement processing can be generated according to the target pixel value.

[0056] In the implementation of the present application, the deep learning method is combined with the mapping table technology to determine the low-illumination enhancement mapping table corresponding to the low-illumination model, and then when the to-be-processed image is received, the low-illumination enhancement mapping table is traversed to obtain the target enhanced image corresponding to the to-be-processed image, which can avoid the problem of complex calculation caused by the deep learning method for low-illumination enhancement of the image, reduce the calculation amount, and improve the efficiency of low-illumination enhancement.

[0057] The determination process of the low-illumination enhancement model provided by the embodiments of the present application is described below Figure 3 In an embodiment, the electronic device determines a pre-trained low-illumination enhancement model, including the following steps:

[0058] S121, obtaining an initial neural network model.

[0059] In an embodiment, the initial neural network model can be a DNN (Deep Neural Network, deep neural network model) or a CNN (Convolutional Neural Network, convolutional neural network model), which is not limited here.

[0060] S122, determining an initial convolution kernel structure corresponding to the initial neural network model.

[0061] In one embodiment, the initial neural network model may include an input layer, a hidden layer, and an output layer. The input layer receives the image to be processed, the hidden layer performs feature processing on the input image to obtain a processed image, and the output layer outputs the processed image. It is understood that the hidden layer in the initial neural network model is crucial for image processing. The hidden layer of the initial neural network model contains an initial convolutional kernel structure with a 3x3 receptive field.

[0062] S123, adjust the initial convolutional kernel structure to the target convolutional kernel structure to obtain the target neural network model.

[0063] The process for determining the target convolutional kernel structure can be found in the following section. Figure 4 Detailed explanation.

[0064] S124: Train the target neural network model using historical low-light images as input data and historical low-light enhancement images as output data to obtain the low-light enhancement model.

[0065] The following combination Figure 4 This application describes the process for determining the target convolutional kernel structure. In one embodiment, the initial convolutional kernel structure includes several convolutional kernels with 3*3 receptive fields. The electronic device adjusts the initial convolutional kernel structure into the target convolutional kernel structure to obtain the target neural network model, including:

[0066] S1231, determine the convolution kernel combination of a single 2*2 receptive field convolution kernel and several 1*1 receptive field convolution kernels.

[0067] Please see Figure 5A , Figure 5A This is a schematic diagram of convolution kernel combinations. Black indicates a 2*2 receptive field convolution kernel, and there is only one of them. White indicates several 1*1 receptive field convolution kernels. The diagram shows four kernels as an example.

[0068] S1232, combine convolution kernels according to the first preset number of combinations to obtain the intermediate convolution kernel structure.

[0069] In one embodiment, the first preset number can be set according to the actual training effect; for example, the first preset number is 4, and there is no limitation here. Please refer to [link to relevant documentation]. Figure 5B , Figure 5B This is a schematic diagram of the intermediate convolutional kernel structure, which is composed of a first preset number of convolutional kernels. The diagram shows an example with a first preset number of 4 kernels. Figure 5B The intermediate convolutional kernel structure shown is used to perform feature processing on the image to be processed, obtaining a feature map, such as... Figure 5B As shown, there are three branches: the first branch, the second branch, and the third branch. The first branch is... Figure 5B The direction of the topmost arrow, the second branch is...Figure 5B The third branch is in the direction of the arrow in the middle. Figure 5B When the image to be processed is input, the first branch, the second branch and the third branch in the combination of a plurality of convolution kernels are used to extract features of the image to be processed to obtain pixel value 1, pixel value 2 and pixel value 3, and then the pixel value 1, the pixel value 2 and the pixel value 3 are averaged to obtain the first pixel value. Figure 5B The "+" symbol is used to indicate the average processing.

[0070] S1233, according to the second preset number, the intermediate convolution kernel structure is stacked to obtain the target convolution kernel structure.

[0071] In an embodiment, as the image illumination complexity increases, a single intermediate convolution kernel structure cannot handle real illumination changes, so the intermediate convolution kernel structure needs to be stacked to meet the image processing of complex illumination. The second preset number can be set according to the actual training effect, for example, the second preset number is 3. Please refer to Figure 5C , Figure 5C The target convolution kernel structure is a schematic diagram of the target convolution kernel structure, which is composed of a second preset number of intermediate convolution kernel structures, and the second preset number is 3 in the figure, which is the first intermediate convolution kernel structure, the second intermediate convolution kernel structure and the third intermediate convolution kernel structure. Figure 5B The intermediate convolution kernel structure is the first intermediate convolution kernel structure, which is used to extract features of the image to be processed to obtain the first pixel value, and a plurality of first pixel values generate the first historical low-illumination feature map. Then, the second historical low-illumination feature map is obtained by the second intermediate convolution kernel structure. Then, the second historical low-illumination feature map is input into the third intermediate convolution kernel structure to obtain the third historical low-illumination feature map (i.e. historical low-illumination enhanced image).

[0072] The present application can increase the receptive field of the model by stacking the intermediate convolution kernel structure, improve the enhancement effect of the model on the image, and set a corresponding mapping table for each intermediate convolution kernel structure. The multi-level intermediate convolution kernel structure can set multiple mapping tables to avoid the problem of long traversal time caused by a single mapping table, and can improve the efficiency of image enhancement.

[0073] S1234, the convolution kernel of the 3*3 receptive field in the initial neural network model is adjusted to the target convolution kernel structure to obtain the target neural network model.

[0074] The following will be combined with Figure 6The training process of the low-illumination enhancement model is described. In an embodiment, the electronic device trains the target neural network model by taking the historical low-illumination image as input data and the historical low-illumination enhanced image as output data, and obtains the low-illumination enhancement model, including:

[0075] In S1241, the historical low-illumination image is processed according to the preset self-integration algorithm to obtain an integrated processed historical low-illumination image.

[0076] In an embodiment, the preset self-integration algorithm refers to an algorithm for rotating the historical low-illumination image. Please refer to Figure 7 , Figure 7 is a schematic diagram of rotating any historical low-illumination image. By rotating the historical low-illumination image (for example, as shown in the four images on the left), each pixel can collect information of 8 surrounding pixels (for example, as shown in the image on the right), which is equivalent to expanding the pixel perception range. Figure 7 Figure 7

[0077] In S1242, the integrated processed historical low-illumination image is input into the target neural network model to obtain a model enhanced image.

[0078] In S1243, the loss value is calculated according to the historical low-illumination enhanced image and the model enhanced image.

[0079] In an embodiment, the loss function corresponding to the historical low-illumination enhanced image and the model enhanced image is determined in advance, and the historical low-illumination enhanced image and the model enhanced image are input into the loss function to obtain the loss value. The loss function can be set according to actual needs, which is not limited here.

[0080] In S1244, when the loss value is less than the preset loss threshold, it is determined that the low-illumination enhancement model is trained.

[0081] The preset loss threshold is a value set in advance to evaluate whether the low-illumination enhancement model is trained. When the loss value is greater than or equal to the preset loss threshold, the model is continuously trained by increasing the training data until the loss value is less than the preset loss threshold.

[0082] The following describes Figure 8 The determination process of the low-illumination enhancement mapping table set is described. In an embodiment, before determining the cascade relationship of a plurality of low-illumination enhancement mapping tables according to the pre-trained low-illumination enhancement model, the electronic device further includes:

[0083] In S131, the convolution kernel calling relationship corresponding to a plurality of intermediate convolution kernel structures in the target convolution kernel structure is determined.

[0084] ​​In an embodiment, the convolution kernel calling relationship refers to the processing logic of a plurality of intermediate convolution kernel structures on an image input to the model. Illustratively, when the number of intermediate convolution kernel structures is 3, they are respectively a first intermediate convolution kernel structure, a second intermediate convolution kernel structure, and a third intermediate convolution kernel structure. When a historical low-illumination image is input to the low-illumination enhancement model, the first intermediate convolution kernel structure is first used to perform feature processing on the historical low-illumination image to obtain a first historical low-illumination feature map; then the second intermediate convolution kernel structure is used to perform feature processing on the first historical low-illumination feature map to obtain a second historical low-illumination feature map; and then the second historical low-illumination feature map is input to the third intermediate convolution kernel structure to obtain a third historical low-illumination feature map (i.e., a historical low-illumination enhanced image). From the above processing logic of the image, it can be seen that the convolution kernel calling relationship is that the first intermediate convolution kernel structure is called by the second intermediate convolution kernel structure, and the second intermediate convolution kernel structure is called by the third intermediate convolution kernel structure.

[0085] In S132, the intermediate convolution kernel structures are called according to the convolution kernel calling relationship to process the historical low-illumination image, to obtain a plurality of historical low-illumination feature maps.

[0086] In an embodiment, after determining that the convolution kernel calling relationship is that the first intermediate convolution kernel structure is called by the second intermediate convolution kernel structure, and the second intermediate convolution kernel structure is called by the third intermediate convolution kernel structure, when a historical low-illumination image is received, the first intermediate convolution kernel structure is first used to perform feature processing on the historical low-illumination image to obtain a first historical low-illumination feature map; then the second intermediate convolution kernel structure is used to perform feature processing on the first historical low-illumination feature map to obtain a second historical low-illumination feature map; and then the second historical low-illumination feature map is input to the third intermediate convolution kernel structure to obtain a third historical low-illumination feature map (i.e., a historical low-illumination enhanced image).

[0087] Please refer to Figure 9 , Figure 9 for a schematic diagram of model processing. Figure 9 In FIG. 1, there are Block1, Block2, …, Block16, wherein Block1 can be understood as Figure 5C the first branch of the first intermediate convolution kernel structure in FIG. 1, Block2 can be understood as Figure 5C the second branch of the first intermediate convolution kernel structure in FIG. 1, Block3 can be understood as Figure 5C the third branch of the first intermediate convolution kernel structure in FIG. 1, Block4 can be understood as Figure 5C the first branch of the second intermediate convolution kernel structure in FIG. 1, Block5 can be understood as Figure 5C the second branch of the second intermediate convolution kernel structure in FIG. 1, and Block6 can be understood as Figure 5CThe third branch of the second intermediate convolution kernel structure. The leftmost side has three 5*5 boxes, and each block in the box represents the pixel value of the image to be processed. There are 3*3 small boxes in the 5*5 box, which can be understood as the receptive field range corresponding to the pixel. The distribution of the three gray pixels in the 3*3 small box can be obtained according to the processing mode of the convolution kernel combination in the first branch, the second branch and the third branch of the first intermediate convolution kernel structure. The initial pixel value of the image to be processed is processed by the first branch, the second branch and the third branch of the first intermediate convolution kernel structure respectively, to obtain pixel value 1, pixel value 2 and pixel value 3. Then, the pixel value 1, the pixel value 2 and the pixel value 3 are averaged to obtain the first pixel value, and a plurality of first pixel values generate a first historical low-illumination feature map. Then, the first historical low-illumination feature map is processed by the first branch, the second branch and the third branch of the second intermediate convolution kernel structure respectively to obtain a second historical low-illumination feature map. In this way, a target enhanced image is obtained.

[0088] In S133, a mapping relationship between the historical low-illumination image and the historical low-illumination feature map and between the historical low-illumination feature maps is determined, and a plurality of low-illumination enhancement mapping tables are constructed according to the mapping relationship.

[0089] In an embodiment, for each intermediate convolution kernel structure, there is a corresponding low-illumination enhancement mapping table. For example, when the number of intermediate convolution kernel structures is 3, the number of low-illumination enhancement mapping tables is also 3, which are a first low-illumination enhancement mapping table, a second low-illumination enhancement mapping table and a third low-illumination enhancement mapping table. The low-illumination enhancement mapping table can include the mapping relationship between the historical low-illumination image and the historical low-illumination feature map, or the mapping relationship between the historical low-illumination feature maps. For example, the first low-illumination enhancement mapping table includes the mapping relationship between the historical low-illumination image and the first historical low-illumination feature map; the second low-illumination enhancement mapping table includes the mapping relationship between the first historical low-illumination feature map and the second historical low-illumination feature map; and the third low-illumination enhancement mapping table includes the mapping relationship between the second historical low-illumination feature map and the third historical low-illumination feature map.

[0090] In S134, the low-illumination enhancement mapping tables are combined to obtain a low-illumination enhancement mapping table set.

[0091] In an embodiment, the above-mentioned three low-illumination enhancement mapping tables are combined to obtain a low-illumination enhancement mapping table set.

[0092] The following will be described in combination with Figure 10 The determination process of the target pixel set provided by the embodiment of the present application is described. In an embodiment, the electronic device traverses a plurality of low-illumination enhancement mapping tables according to a cascade relationship to obtain a target pixel set corresponding to an initial pixel set, comprising:

[0093] S151, determining a first low-illumination enhancement mapping table and a corresponding next low-illumination enhancement mapping table from the plurality of low-illumination enhancement mapping tables according to the cascade relationship.

[0094] In an embodiment, the first low-illumination enhancement mapping table refers to the first mapping table traversed when receiving the image to be processed, and the next low-illumination enhancement mapping table refers to the mapping table traversed after the first mapping table. When the number of low-illumination enhancement mapping tables is three, which are the first low-illumination enhancement mapping table, the second low-illumination enhancement mapping table and the third low-illumination enhancement mapping table, the first low-illumination enhancement mapping table is the first low-illumination enhancement mapping table, the next low-illumination enhancement mapping table is the second low-illumination enhancement mapping table, and the low-illumination enhancement mapping table of the next level is the third low-illumination enhancement mapping table.

[0095] S152, obtaining an initial pixel value corresponding to each initial pixel in the initial pixel set, and traversing the first low-illumination enhancement mapping table according to each initial pixel value in the initial pixel set to obtain a first pixel value.

[0096] In an embodiment, each initial pixel corresponds to a luminance value of the pixel (i.e., the initial pixel value). For example, the luminance value of the pixel of the image to be processed is set in the interval [0, 255], wherein the closer to 255, the higher the luminance of the pixel, and the closer to 0, the lower the luminance of the pixel. Traversing the first low-illumination enhancement mapping table (i.e., the first low-illumination enhancement mapping table) according to each initial pixel value in the initial pixel set can obtain the first pixel value.

[0097] S153, taking the first pixel value obtained by traversing the first level as the input of the next level, and continuing to traverse the next low-illumination enhancement mapping table according to the first pixel value until the target pixel value corresponding to each initial pixel value is obtained.

[0098] In an embodiment, taking the first pixel value as the input of the next level, traversing the second low-illumination enhancement mapping table according to the first pixel value to obtain a second pixel value; then taking the second pixel value as the input of the next level, traversing the second low-illumination enhancement mapping table according to the second pixel value to obtain the target pixel value corresponding to each initial pixel value.

[0099] S154, combining the target pixel values to obtain a target pixel set.

[0100] Please refer to Figure 11 , Figure 11is a structural schematic diagram of a low-illumination image processing apparatus provided in an embodiment of the present application. In some embodiments, the low-illumination image processing apparatus 20 can include a plurality of functional modules composed of computer program segments. The computer programs of each program segment in the low-illumination image processing apparatus 20 can be stored in a memory of an electronic device and executed by at least one controller to perform the functions of low-illumination enhancement processing (see detailed description) Figure 2 .

[0101] In the present embodiment, the low-illumination image processing apparatus 20 can be divided into a plurality of functional modules according to the functions performed thereby. The functional modules can include an image receiving module 201, a mapping table determining module 202, a mapping table traversing module 203, and an image enhancement module 204. The term "module" as used herein refers to a series of computer program segments that can be executed by at least one controller and can complete a fixed function, which are stored in a memory. In the present embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0102] The image receiving module 201 can be configured to, when receiving a to-be-processed image, determine a plurality of initial pixels corresponding to the to-be-processed image to obtain an initial pixel set.

[0103] The mapping table determining module 202 can be configured to determine a cascade relationship of a plurality of low-illumination enhancement mapping tables according to a pre-trained low-illumination enhancement model, wherein the low-illumination enhancement mapping table records a corresponding relationship of pixel values.

[0104] The mapping table traversing module 203 can be configured to traverse the plurality of low-illumination enhancement mapping tables according to the cascade relationship to obtain a target pixel set corresponding to the initial pixel set.

[0105] The image enhancement module 204 can be configured to generate a target enhanced image corresponding to the to-be-processed image according to the target pixel set.

[0106] In the embodiments of the present application, the functional modules in the low-illumination image processing apparatus 20 belong to the same application concept as the low-illumination image processing method in the above-described embodiments, and the specific implementation modes of the modules of the low-illumination image processing apparatus 20 correspond to the steps of the low-illumination image processing method in the above-described embodiments, which will not be described herein again.

[0107] Referring to Figure 12 , a structural schematic diagram of an electronic device provided in an embodiment of the present application is shown. In the preferred embodiments of the present application, the electronic device 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0108] Those skilled in the art should understand that Figure 12The structure of the electronic device shown does not constitute a limitation of the embodiments of the present application, and can be a bus structure or a star structure. The electronic device 3 can also include more or fewer other hardware or software, or different component arrangements than those shown.

[0109] In some embodiments, the electronic device 3 is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. The hardware of the electronic device 3 includes, but is not limited to, a microprocessor, an application-specific integrated circuit, a programmable gate array, a digital processor, and an embedded device. The electronic device 3 can also include a client device, which includes, but is not limited to, any electronic product that can interact with a client through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, or the like, such as a personal computer, a tablet computer, a smart phone, a digital camera, and the like.

[0110] It should be noted that the electronic device 3 is only an example, and other existing or future electronic products, such as those that can be adapted to the present application, should also be included in the protection scope of the present application and are hereby incorporated by reference.

[0111] In some embodiments, the memory 31 stores a computer program that, when executed by the at least one processor 32, implements all or part of the steps in the low-illumination image processing method. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk memory, a magnetic disk memory, a magnetic tape memory, or any other computer-readable medium capable of carrying or storing data.

[0112] Further, the computer-readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, and the like; and the data storage area can store created data and the like.

[0113] In some embodiments, the at least one processor 32 is a control unit of the electronic device 3, which connects various components of the entire electronic device 3 through various interfaces and lines, and performs various functions of the electronic device 3 and processes data by running or executing programs or modules stored in the memory 31 and calling data stored in the memory 31. For example, the at least one processor 32 implements all or part of the steps of the low-illumination image processing described in the embodiments of the present application when executing the computer program stored in the memory, or implements all or part of the functions of the low-illumination image processing apparatus. The at least one processor 32 can be composed of integrated circuits, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0114] In some embodiments, the at least one communication bus 33 is configured to realize the connection and communication between the memory 31, the at least one processor 32, and the like.

[0115] Although not shown, the electronic device 3 can further include a power supply (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the at least one processor 32 through a power management device, so as to realize the functions of managing charging, discharging, and power consumption management, and the like through the power management device. The power supply can further include one or more direct-current or alternating-current power supplies, recharging devices, power supply fault detection circuits, power supply converters or inverters, power supply status indicators, and the like. The electronic device 3 can further include various sensors, Bluetooth modules, Wi-Fi modules, and the like, which are not described here.

[0116] The integrated units in the form of software function modules described above can be stored in a computer-readable storage medium. The software function modules described above are stored in a storage medium, and include a plurality of instructions for causing an electronic device (which can be a personal computer, an electronic device, or a network device, etc.) or a processor to execute part of the method described in each embodiment of the present application.

[0117] In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the modules is only a logical function division. There can be another division manner in actual implementation.

[0118] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, and may be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0119] In addition, the functional modules in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional modules.

[0120] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims. In addition, it is obvious that the word "comprising" does not exclude other units or, singular does not exclude plural. The plurality of units or devices stated in the specification can also be realized by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, not any specific order.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A low-illumination image processing method, characterized by, The method comprises: When receiving a to-be-processed image, a plurality of initial pixels corresponding to the to-be-processed image are determined, and an initial pixel set is obtained; A plurality of intermediate convolution kernel structures corresponding to a convolution kernel calling relationship in a target convolution kernel structure are determined; According to the convolution kernel calling relationship, the intermediate convolution kernel structures are called respectively to process historical low-illumination images, and a plurality of historical low-illumination feature maps are obtained; A mapping relationship of pixel values between the historical low-illumination images and the historical low-illumination feature maps and between the historical low-illumination feature maps is determined, and a plurality of low-illumination enhancement mapping tables are constructed according to the mapping relationship; The plurality of low-illumination enhancement mapping tables are combined to obtain a low-illumination enhancement mapping table set; According to a pre-trained low-illumination enhancement model, a cascade relationship of the plurality of low-illumination enhancement mapping tables is determined, comprising: determining the intermediate convolution kernel structure corresponding to each low-illumination enhancement mapping table; determining the cascade relationship of the plurality of low-illumination enhancement mapping tables according to the convolution kernel calling relationship of the intermediate convolution kernel structure; wherein the low-illumination enhancement mapping table records the corresponding relationship of pixel values, and the training method of the low-illumination enhancement model comprises: obtaining an initial neural network model; determining an initial convolution kernel structure corresponding to the initial neural network model; adjusting the initial convolution kernel structure to the target convolution kernel structure to obtain a target neural network model; training the target neural network model with the historical low-illumination images as input data and historical low-illumination enhancement images as output data to obtain the low-illumination enhancement model; According to the cascade relationship, the plurality of low-illumination enhancement mapping tables are traversed to obtain a target pixel set corresponding to the initial pixel set; According to the target pixel set, a target enhancement image corresponding to the to-be-processed image is generated.

2. The method of claim 1, wherein, The initial convolution kernel structure includes a plurality of The convolution kernel of the receptive field, adjusting the initial convolution kernel structure to a target convolution kernel structure, obtaining a target neural network model, includes: determining individual a convolution kernel of a receptive field and a number of a combination of convolution kernels of a convolution kernel of a receptive field; The plurality of convolution kernel combinations are combined according to a first preset number to obtain an intermediate convolution kernel structure; The intermediate convolution kernel structure is stacked according to a second preset number to obtain a target convolution kernel structure; The initial neural network model is adjusted to obtain a target neural network model. The convolution kernel of the receptive field is adjusted to the target convolution kernel structure to obtain the target neural network model.

3. The method of claim 1, wherein, According to the cascade relationship, the plurality of low-illumination enhancement mapping tables are traversed to obtain a target pixel set corresponding to the initial pixel set, comprising: According to the cascade relationship, a first-level low-illumination enhancement mapping table and a corresponding next-level low-illumination enhancement mapping table are determined from the plurality of low-illumination enhancement mapping tables; The initial pixel value corresponding to each initial pixel in the initial pixel set is obtained, and the first-level low-illumination enhancement mapping table is traversed according to each initial pixel value in the initial pixel set to obtain a first pixel value; The first pixel value obtained by the first-level traversal is taken as the input of the next-level traversal, and the next-level low-illumination enhancement mapping table is continued to be traversed according to the first pixel value, until the target pixel value corresponding to each initial pixel value is obtained; The target pixel values are combined to obtain a target pixel set.

4. The method of claim 1, wherein, According to a preset self-integration algorithm, the historical low-illumination images are processed to obtain integrated historical low-illumination images; ​ input the integrated historical low-illumination image to the target neural network model to obtain a model enhanced image; calculate a loss value according to the historical low-illumination enhanced image and the model enhanced image; when the loss value is less than a preset loss threshold, use the target neural network model as the pre-trained low-illumination enhancement model.

5. A low-illuminance image processing apparatus for executing the low-illuminance image processing method according to any one of claims 1 to 4, characterized by The device comprises: An image receiving module is configured to, when receiving a to-be-processed image, determine a plurality of initial pixels corresponding to the to-be-processed image to obtain an initial pixel set. A mapping table determining module is configured to determine a cascade relationship of a plurality of low-illumination enhancement mapping tables according to a pre-trained low-illumination enhancement model, wherein the low-illumination enhancement mapping table records a corresponding relationship of pixel values. A mapping table traversing module is configured to traverse the plurality of low-illumination enhancement mapping tables according to the cascade relationship to obtain a target pixel set corresponding to the initial pixel set. An image enhancement module is configured to generate a target enhanced image corresponding to the to-be-processed image according to the target pixel set.

6. An electronic device, comprising: The electronic device comprises a processor configured to implement the low-illumination image processing method according to any one of claims 1 to 4 when executing a computer program stored in a memory.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is configured to implement the low-illumination image processing method according to any one of claims 1 to 4 when executed by a controller.

Images