All-day image low-light enhancement and semantic segmentation joint method and system

Abstract

The application discloses a method and system suitable for all-day image low-illumination enhancement and semantic segmentation combination, and belongs to the technical field of computer vision. The application firstly combines low-illumination enhancement and semantic segmentation together, the whole network adopts an end-to-end training mode, the enhancement task and the segmentation task learn some common low-level features by sharing the bottom layer, share information, complement each other, and improve the performance of both; by jointly optimizing the enhancement network and the semantic segmentation network, the result of the image enhancement of the enhancement network will be reconstructed under the guidance of the downstream feature extraction network and the information of the two task branches, and at the same time, the semantic segmentation network learns the structural detail features and color features recovered in the low-illumination image enhancement, thereby improving the accuracy of the semantic segmentation network; the low-light image can be enhanced to assist the driver to perceive the low-light environment, and the semantic segmentation result can be obtained from the perspective of the visual task, and the application has strong adaptability to night data and daytime data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and image processing, and more particularly relates to a method and system for image low-light enhancement and semantic segmentation suitable for all-day. BACKGROUND

[0002] In recent years, with the development of deep learning, artificial intelligence technology has been increasingly applied in the field of automatic driving. One of the goals of automatic driving technology is to reduce the occurrence of traffic accidents. Currently, a considerable number of traffic accidents are caused by human driver's operation errors, and the use of automatic driving systems instead of human drivers can effectively reduce the occurrence of accidents. Night is a high-incidence scene of traffic accidents, and there is a greater demand for automatic driving technology. Therefore, low-light enhancement and accurate semantic segmentation of city road images in night scenes have become a current research hotspot.

[0003] Low-light image enhancement is one of the core problems of image processing technology, and can be widely applied in high-level computer vision tasks such as semantic segmentation, target recognition, and target detection. Due to the limitations of existing visual imaging technology, under low-light conditions such as night, insufficient light illumination leads to serious degradation of the images collected by the imaging sensor. Low-light enhancement technology can adjust the low-light images to obtain enhanced images, thereby greatly improving human perception of low-light environments and assisting in high-level vision tasks such as semantic segmentation, target recognition, and target detection.

[0004] Semantic segmentation is one of the core problems in the field of automatic driving. Semantic segmentation can divide each pixel in the collected road scene image into the corresponding category in real time, realize the classification of the road scene image at the pixel level, and understand the semantic information of the surrounding environment, which has very important practical significance for avoiding obstacles and reducing collisions between vehicles and people.

[0005] Deep neural networks have achieved the most advanced level in low-light image enhancement and semantic segmentation tasks. Compared with traditional methods, deep neural networks can learn better feature representations, thereby achieving outstanding results.

[0006] However, current low-light enhancement algorithms are mainly designed for human vision rather than high-level vision tasks. Although some methods can improve the overall visual quality and facilitate human observation of images, they may damage the high-level semantic information of the images, which is not conducive to high-level vision tasks such as semantic segmentation and target detection. At the same time, current semantic segmentation algorithms are often only suitable for daytime scenes or only for night scenes, and lack a solution that is suitable for both daytime scenes and night scenes. SUMMARY

[0007] In view of the defects of the prior art, the present application aims to provide a method and system for image low-illumination enhancement and semantic segmentation suitable for all day, aiming to solve the problem that the existing semantic segmentation scheme is basically only for single scene data of daytime or nighttime.

[0008] To achieve the above-mentioned purpose, in a first aspect, the present application provides a method for image low-illumination enhancement and semantic segmentation suitable for all day, which comprises:

[0009] inputting a normal-illumination image to be processed into the trained multi-task joint model to obtain a pixel-level semantic segmentation result; or inputting a low-illumination image to be processed into the trained multi-task joint model to obtain a pixel-level semantic segmentation result and a 3-channel reflectance map after low-illumination enhancement;

[0010] The normal-illumination image is an image taken under the condition that the environmental illumination intensity is greater than or equal to 1 Lux, and the low-illumination image is an image taken under the condition that the environmental illumination intensity is less than 1 Lux.

[0011] The multi-task joint model adopts an end-to-end training mode, and specifically comprises:

[0012] a light intensity enhancement network, which is used for not processing the input normal-illumination image in any way and directly outputting to a feature extraction network, and performing light intensity enhancement on the input low-illumination image to obtain a 3-channel reflectance map and a 1-channel brightness map and output to the feature extraction network;

[0013] a feature extraction network, which is used for extracting a semantic feature map from the normal-illumination image or the 3-channel reflectance map and outputting to a semantic segmentation module, and extracting a low-level feature from the 3-channel reflectance map and outputting to a low-illumination enhancement module at the same time as the extracted semantic feature map, wherein the low-level feature comprises a spatial feature, a texture feature and a color feature;

[0014] a low-illumination enhancement module, which is used for up-sampling the semantic feature map, then fusing and decoding after the low-level feature to obtain an enhanced 3-channel reflectance map and a 1-channel brightness map;

[0015] a semantic segmentation module, which is used for performing pixel-level semantic segmentation on the input semantic feature map to obtain the class of each pixel of the input image.

[0016] Preferably, the light intensity enhancement network comprises:

[0017] a 9*9 convolution layer, a 3*3 first convolution layer and a first Relu activation layer, a 3*3 second convolution layer with a step of 2 and a second Relu activation layer, a 3*3 third convolution layer and a third Relu activation layer, a 3*3 fourth de-convolution layer with a 2-fold up-sampling and a fourth Relu activation layer, a 3*3 fifth convolution layer and a fifth Relu activation layer, a 3*3 sixth convolution layer, a 3*3 seventh convolution layer, and a Sigmoid layer connected in sequence;

[0018] wherein the input of the 9*9 convolution layer is the input low-illumination image, the first convolution layer and the first Relu activation layer, the second convolution layer and the second Relu activation layer, the third convolution layer and the third Relu activation layer, the fourth de-convolution layer and the fourth Relu activation layer, the fifth Relu activation layer, the seventh convolution layer, and the Sigmoid layer are all connected in series, and the input is the output of the former;

[0019] the input of the fifth convolution layer is the splicing of the output of the fourth Relu activation layer and the output of the first Relu activation layer;

[0020] the input of the sixth convolution layer is the splicing of the output of the fifth Relu activation layer and the output of the 3*3 eighth convolution layer on the input low-illumination image;

[0021] the Sigmoid layer outputs a 3-channel reflectance map and a 1-channel brightness map.

[0022] It should be noted that the light intensity enhancement network preferably has the above structure, different levels of features are extracted through the convolution layers connected in series, the low-illumination image can be effectively decomposed into a reflectance map and a brightness map, the network is trained in a self-supervised manner, does not require paired data, has a small number of parameters, and occupies less resources during running.

[0023] Preferably, the feature extraction network comprises:

[0024] a 3*3 convolution layer for extracting low-level features from the 3-channel reflectance map;

[0025] a swin transformer encoder for extracting semantic features from the 3-channel reflectance map or a normal illumination image, and outputting the semantic features to an upernet decoder;

[0026] the upernet decoder is used for preliminarily decoding the semantic features to obtain a 1 / 4 resolution feature map of the original image.

[0027] It should be noted that the feature extraction network of the preferred structure of the present application can effectively extract semantic features required by the semantic segmentation branch and semantic features required by the enhancement branch, the upnet decoder can effectively decode the semantic features to obtain new feature maps conducive to subsequent tasks, and the low-level features extracted by the 3*3 convolution layer can supplement spatial texture and other detailed feature information for the enhancement branch.

[0028] Preferably, the low-light enhancement module comprises:

[0029] a 3*3 first deconvolution layer, a first activation layer, a 3*3 second deconvolution layer and a second activation layer connected in sequence, for upsampling the semantic feature map;

[0030] a concatenation layer for concatenating the low-level features and the upsampled semantic features to obtain a concatenated feature map;

[0031] a 3*3 convolution layer and a Sigmoid connected in sequence, for decoding the concatenated feature map to obtain an enhanced 3-channel reflectance map and a 1-channel luminance map, the 3-channel reflectance map serving as a low-light enhancement result.

[0032] It should be noted that the low-light enhancement module of the preferred structure of the present application can decode the 1 / 4 resolution feature map with a small amount of parameters, and at the same time of decoding, realize information fusion with the low-level features to obtain a reflectance map and a luminance map, wherein the reflectance map can serve as a low-light image enhancement result map.

[0033] Preferably, the semantic segmentation module comprises a 3*3 first deconvolution layer, a first activation layer, a 3*3 second deconvolution layer and a second activation layer connected in sequence, for decoding the 1 / 4 resolution feature map to obtain a semantic segmentation result map containing the class of each pixel point.

[0034] It should be noted that the semantic segmentation module of the preferred structure of the present application can effectively decode the semantic segmentation result map of the original image from the 1 / 4 resolution feature map with a small amount of parameters, i.e. the semantic segmentation result of each pixel class in the original image.

[0035] Preferably, the total loss function calculation formula during training is as follows:

[0036] L 总 =L1+L2+L3

[0037] L1=L re1 +αL R1 +βL I1

[0038] L2=L re2 +υL R2 +νLI2

[0039]

[0040] wherein, L1 represents the image enhancement loss of the light intensity enhancement network, L2 represents the image enhancement loss of the low-illumination enhancement module, L3 represents the image semantic segmentation loss of the semantic segmentation module, L re1 R1 I1 respectively represent the reconstruction loss, the reflection loss and the illumination component smoothing loss of the light intensity enhancement network, L re2 R2 I2 respectively represent the reconstruction loss, the reflection loss and the illumination component smoothing loss of the low-illumination enhancement module, and α, β, υ, ν represent weight parameters, M represents the number of categories, y c represents the category judgment of a variable, and is 1 if the categories are the same and is 0 if the categories are different, p c represents the predicted probability that the observed sample belongs to the category c.

[0041] It should be noted that the total loss function in the present application covers the semantic segmentation loss and the low-illumination enhancement loss, wherein the low-illumination enhancement loss further covers the reconstruction loss, the reflection map loss and the brightness map smoothing loss. The semantic segmentation loss is the prediction error of semantic segmentation, which is used to guide the network training to achieve better semantic segmentation effect. In the low-illumination enhancement loss, the reconstruction loss is used to ensure that the decomposed reflection map and brightness map conform to the original image information; in the reflection map loss, it is considered that the histogram equalization of F(X) on the image can greatly improve the information entropy of the image, so the reflection map loss ensures that the maximum channel of the reflection map conforms to the maximum channel of the low-light image and has the maximum entropy, so that the reflection map can better restore the original image information and achieve better visual effect; the brightness map smoothing loss is used to remove noise and restore local details in the process of obtaining the brightness map by the enhancement network.

[0042] Preferably, the training process uses the normal illumination semantic segmentation dataset, the low-illumination semantic segmentation dataset and the low-illumination image enhancement dataset together for training, and the specific manner comprises:

[0043] In each batch of iterations in the training process, there are fixed 3 input pictures, which are respectively 3 pictures from cityscapes, nightcity and darkface, representing the normal illumination semantic segmentation dataset, the low-illumination semantic segmentation dataset and the low-illumination image enhancement dataset respectively;

[0044] The cityscapes dataset data only needs to obtain the semantic segmentation result map after passing through the network without low-illumination enhancement, and the semantic segmentation result map is used to calculate the semantic segmentation task loss; ​​​​

[0045] The nightcity dataset data obtains a semantic segmentation result image and a low-illumination enhancement result image through the network at the same time, the semantic segmentation result image is used to calculate the semantic segmentation task loss, and the low-illumination enhancement result image is used to calculate the low-illumination enhancement loss.

[0046] The darkface dataset data obtains a low-illumination enhancement result through the network, and is used to calculate the low-illumination enhancement loss.

[0047] It should be noted that the application preferably trains the above-mentioned dataset selection mode, because the characteristics of the multi-task model enable the network to simultaneously input low-illumination dataset and semantic segmentation dataset for training, and simultaneously optimize the low-illumination image enhancement task and the semantic segmentation task; secondly, training using multiple different types of datasets can improve the generalization of the model, so that the network can achieve good results in the semantic segmentation task of normal illumination data and low-illumination data and the low-illumination enhancement task of low-illumination data; and simultaneously training multiple different types of datasets can better optimize the final multi-task target of the model.

[0048] To achieve the above object, in a second aspect, the application provides a full-day image low-illumination enhancement and semantic segmentation joint system, comprising: a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, so that the method of the first aspect is executed.

[0049] Overall, compared with the prior art, the above technical scheme conceived by the application has the following beneficial effects:

[0050] The application proposes a full-day image low-illumination enhancement and semantic segmentation joint method and system, which first combines the low-illumination enhancement task and the semantic segmentation task together, the entire network adopts an end-to-end training mode, the enhancement task and the segmentation task learn some common low-level features by sharing some bottom layers, and the performance of the two is improved by sharing information and complementing each other; by jointly optimizing the enhancement network and the semantic segmentation network, the result of image enhancement by the enhancement network is reconstructed under the guidance of the downstream feature extraction network and the information of the two task branches, and the semantic segmentation network can learn the structural detail features and color features restored in the low-illumination image enhancement, thereby improving the accuracy of the semantic segmentation network; the low-light image can be enhanced to assist the driver in perceiving the low-light environment, and the semantic segmentation result can be obtained at the high-level visual task level, and the system has strong adaptability to night data and daytime data. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 It is a full-day image low-illumination enhancement and semantic segmentation joint method flowchart provided by the application.

[0052] Figure 2 is a schematic diagram of the light enhancement network structure provided by the present application.

[0053] Figure 3 is a schematic diagram of the feature extraction network structure provided by the present application.

[0054] Figure 4 is a schematic diagram of the low-light enhancement module structure provided by the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0056] The present application provides a method for image low-light enhancement and semantic segmentation suitable for all day, which specifically comprises the following steps:

[0057] Obtain normal light urban road data set and night urban road data set.

[0058] Construct a light enhancement network, which is a convolutional network. The function of the light enhancement network is to enhance low-light images. At the same time, the present application can also adapt to normal light data. For normal light images, the present application sets a constraint that the normal light images remain unchanged after passing through the enhancement network. For low-light images, the obtained enhanced images provide prior information for subsequent semantic segmentation tasks and further low-light enhancement.

[0059] Construct a feature extraction network, which uses swin transformer as the backbone network and upnet as the decoder to obtain a feature map with a resolution of 1 / 4 of the original image. Specifically, the present application improves the network structure. In order to help the subsequent enhancement task, the present application adds a 3*3 convolutional layer before the swin transformer network to extract low-level features. The low-level features contain more spatial texture and other detailed information. Therefore, sending the features to the subsequent enhancement task branch for feature fusion can obtain low-light enhancement images with better restoration effect.

[0060] The 1 / 4 feature map obtained by the feature extraction network is subjected to two different upsampling branches, and finally a low-illumination enhancement map and a semantic segmentation result map are obtained. The semantic segmentation branch is composed of two 3*3 deconvolution layers, and the low-illumination enhancement branch also contains two 3*3 deconvolution layers. The feature maps obtained after the two deconvolution layers are fused with the feature maps obtained by the first 3*3 convolution layer of the feature extraction network, and then a convolution layer is used to obtain the final enhancement map. By setting the two-task branch module, the low-illumination enhancement map and the semantic segmentation result map can be obtained at the same time.

[0061] Finally, the low-illumination semantic segmentation model based on multi-task learning is used to perform low-illumination enhancement and semantic segmentation on the input image to obtain a semantic segmentation result map and a low-illumination enhancement map.

[0062] The entire network adopts an end-to-end training method, and the illumination enhancement network and the semantic segmentation network are jointly optimized. The results of the enhancement network are reconstructed under the guidance of the downstream feature extraction network and the two-task branch information, and the semantic segmentation network can learn the structural detail features and color features recovered in the low-illumination image enhancement, thereby improving the accuracy of the semantic segmentation network.

[0063] Since it is a multi-task learning method, during the training process, the present application can simultaneously use multiple data sets such as normal illumination image data sets and low-illumination image data sets for network training, thereby achieving the effect of improving the generalization of the network, so that the trained network can be applicable to low-illumination images and normal-illumination images at the same time, and the multi-data set training is helpful to improve the effects of the enhancement task and the semantic segmentation task.

[0064] As shown in Figure 1 The entire architecture is composed of three parts: 1) a self-supervised enhancement network based on Retinex decomposition theory; 2) a feature extraction network; and 3) a low-illumination enhancement task and a semantic segmentation task two-task branch.

[0065] The first part is to obtain night image enhancement through the enhancement network. It takes a low-illumination image as input and decomposes it into a reflection map and a brightness map, and takes the reflection map as the result of night image enhancement. If it is a normal-illumination image, no enhancement is needed.

[0066] The second part is to combine the enhanced image obtained by the enhancement network, use the swintransformer with good performance in the semantic segmentation field as the encoder, and use the upernet as the decoder to perform image feature extraction and obtain a 1 / 4 resolution feature map.

[0067] The third part is to decode the feature map through two separate branches to obtain the night image enhancement result and the semantic segmentation result image respectively, and if it is a normal illumination image, the low-illumination enhancement result is not needed, and only the semantic segmentation result image is needed.

[0068] Next, the network structure of each part will be described in detail.

[0069] The enhancement network is based on the Retinex theory. The classic Retinex theory establishes a human color perception model. This theory assumes that the image observed by the human eye can be decomposed into two parts, the reflection channel and the illumination channel. Assuming that S represents the source image, the expression S = R ° I can be obtained, wherein R represents the reflection component, I represents the illumination component, and ° represents element-wise multiplication.

[0070] As shown in Figure 2 The light intensity enhancement network includes, in series: a 9*9 convolution layer, a 3*3 first convolution layer and a first Relu activation layer, a 3*3 second convolution layer with a step of 2 and a second Relu activation layer, a 3*3 third convolution layer and a third Relu activation layer, a 3*3 fourth deconvolution layer with a 2-fold upsampling and a fourth Relu activation layer, a 3*3 fifth convolution layer and a fifth Relu activation layer, a 3*3 sixth convolution layer, a 3*3 seventh convolution layer, and a Sigmoid layer. The input of the 9*9 convolution layer is the input low-illumination image. The inputs of the first convolution layer and the first Relu activation layer, the second convolution layer and the second Relu activation layer, the third convolution layer and the third Relu activation layer, the fourth deconvolution layer and the fourth Relu activation layer, the fifth Relu activation layer, the seventh convolution layer, and the Sigmoid layer are all the outputs of the former in the series. The input of the fifth convolution layer is the splicing of the output of the fourth Relu activation layer and the output of the first Relu activation layer. The input of the sixth convolution layer is the splicing of the output of the fifth Relu activation layer and the output of the 3*3 eighth convolution layer on the input low-illumination image. The Sigmoid layer outputs a 3-channel reflection map and a 1-channel brightness map.

[0071] Since it is difficult to obtain corresponding normal illumination label data for night city road data, it is difficult to use supervised learning, so the present application uses a self-supervised training method, and a non-reference loss function independent of label data is selected as the loss function. In order to ensure that the image after recovering the illumination can retain the object edge information while retaining the smooth transition of the illumination information, the following loss function is used in the illumination enhancement network:

[0072]

[0073] Wherein, L re represents the reconstruction loss, S represents the low-illumination input image, R represents the reflection map, and I represents the brightness map.

[0074]

[0075] where L R denotes the reflection loss, F(X) denotes the histogram equalization operator of image X. λ is the weight parameter, and ΔR denotes the gradient image of the reflection map. This loss function means that the maximum channel of reflectance should conform to the maximum channel of the low-light image and have the maximum entropy.

[0076]

[0077] where L I denotes the illumination component smoothness loss.

[0078] The total loss function of low-light enhancement is:

[0079]

[0080] where λ1, λ2, λ3, λ4 are weight parameters.

[0081] Secondly, the feature extraction network, such as Figure 3 As shown in the figure, the current effect is better swin transformer, a deep learning model based on transformer, has good performance in visual tasks, and is used as the backbone of many visual model architectures today. Since the network needs to build two task branches of low-light enhancement and semantic segmentation at the end, the low-light enhancement branch has greater demand for spatial information contained in low-level features, therefore the network first uses a 3*3 convolution to extract low-level features from the input low-light image and its enhanced image (normal illumination image does not need to be enhanced or uses the original image), at the same time, the low-light image and the enhanced image are also sent into the swin transformer network, which is the same as the original swin transformer network structure, and finally encodes the input data to get a feature map with a resolution of 1 / 32 of the original image, then uses the upnet decoder to decode the feature map to a feature map with a resolution of 1 / 4 of the original image, and sends the feature map to the subsequent low-light enhancement task branch and semantic segmentation task branch for decoding.

[0082] Finally, the low-light enhancement and semantic segmentation two task branches are explained, as shown in the figure, Figure 1 After the feature extraction network in the last step has obtained the 1 / 4 feature map, it is finally obtained through two different up-sampling branches to obtain the low-light enhancement image and the semantic segmentation result image.

[0083] The semantic segmentation branch consists of two 3*3 deconvolutional layers and an activation layer. After passing through these two deconvolutional layers, 1 / 4 of the feature map is used to obtain the final semantic segmentation result map.

[0084] like Figure 4 As shown, the low-light enhancement branch also contains two 3x3 deconvolution layers and an activation layer. After passing through these two deconvolution layers, the 1 / 4 feature map becomes a feature map with the same resolution as the original image. This feature map is then concatenated with the low-level feature map obtained from the first convolutional layer of the feature extraction network along the channel dimension. A further 3x3 convolutional layer is then used to fuse the features. Finally, similar to the enhancement network built on Retinex decomposition theory, a sigmoid layer is used to obtain the final reflection and brightness maps, where the reflection map is the restored image. The loss function of the low-light enhancement branch is the same as that of the enhancement network.

[0085] The semantic segmentation branch uses the cross-entropy loss function:

[0086]

[0087] Where M represents the number of categories, y c This indicates the category judgment of the variable; 1 represents the same category, and 0 represents different categories. c This represents the predicted probability that the observed sample belongs to category c.

[0088] The training process of this invention uses both normal lighting datasets and low lighting datasets for training, with two objectives:

[0089] Training with multiple different types of datasets can improve the generalization of the model, enabling the network to achieve good results in both normal lighting and low lighting data segmentation tasks and low-light image enhancement tasks.

[0090] Secondly, training with multiple different types of datasets simultaneously can better optimize the model's final multi-task objectives. In this invention, this means optimizing the low-light enhancement task objective and the semantic segmentation task objective.

[0091] Training on multiple datasets can be done in two ways: sequentially using different datasets (e.g., training on dataset A first, then training on dataset B after A is complete); or simultaneously using data from datasets A and B within a single batch. To avoid biasing the model training towards a particular task or dataset, we chose to train using data from multiple datasets simultaneously.

[0092] The training manner adopted by the present application is that three input pictures in a batch are fixed, which are three pictures from cityscapes, nightcity and darkface respectively, wherein the cityscapes dataset data only needs to obtain a semantic segmentation result graph after passing through the network without low-illumination enhancement, and the semantic segmentation result graph is used to calculate a semantic segmentation task loss; the nightcity dataset data obtains a semantic segmentation result graph and a low-illumination enhancement result graph through the network, the semantic segmentation result graph is used to calculate a semantic segmentation task loss, and the low-illumination enhancement result graph is used to calculate a low-illumination enhancement loss; the darkface dataset data obtains a low-illumination enhancement result through the network, and is used to calculate a low-illumination enhancement loss.

[0093] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for all-day image low-light enhancement and semantic segmentation combination, characterized in that, The method comprises: inputting a normal light image to be processed into the trained multi-task joint model to obtain a pixel-level semantic segmentation result; or inputting a low-light image to be processed into the trained multi-task joint model to obtain a pixel-level semantic segmentation result and a 3-channel reflection map after low-light enhancement as an enhanced result; the normal light image is an image captured under the condition that the environmental light intensity is greater than or equal to 1 Lux, and the low-light image is an image captured under the condition that the environmental light intensity is less than 1 Lux; The multi-task joint model adopts an end-to-end training mode, and specifically comprises: a light intensity enhancement network, which is used for not processing the input normal light image and directly outputting to a feature extraction network, and performing light intensity enhancement on the input low-light image to obtain a 3-channel reflection map and a 1-channel brightness map and output to the feature extraction network; a feature extraction network, which is used for extracting a semantic feature map from the normal light image or the 3-channel reflection map and outputting to a semantic segmentation module, and extracting a low-level feature from the 3-channel reflection map and outputting to a low-light enhancement module at the same time, the low-level feature comprising a spatial feature, a texture feature and a color feature; a low-light enhancement module, which is used for upsampling the semantic feature map, fusing the upsampled semantic feature map with the low-level feature, and decoding to obtain the enhanced 3-channel reflection map and the 1-channel brightness map; a semantic segmentation module, which is used for performing pixel-level segmentation on the input semantic feature map to obtain the category of each pixel of the input image; The light intensity enhancement network comprises: Serial Convolutional layers The first convolutional layer and the first ReLU activation layer, with a stride of 2. The second convolutional layer and the second ReLU activation layer The third convolutional layer and the third ReLU activation layer, with 2x upsampling The fourth deconvolutional layer and the fourth ReLU activation layer The fifth convolutional layer and the fifth ReLU activation layer, sixth convolutional layer Seventh convolutional layer, Sigmoid layer; wherein, The input of the convolutional layer is the input low-illumination image, the first convolutional layer and the first Relu activation layer, the second convolutional layer and the second Relu activation layer, the third convolutional layer and the third Relu activation layer, the fourth deconvolutional layer and the fourth Relu activation layer, the fifth Relu activation layer, the seventh convolutional layer, and the Sigmoid layer, all of which are in a serial relationship. the input of the fifth convolutional layer is the splicing of the output of the fourth Relu activation layer and the output of the first Relu activation layer; The input of the sixth convolutional layer is the output of the fifth Relu activation layer and the low light intensity image inputted through The splicing of the output of the eighth convolutional layer; the Sigmoid layer outputs a 3-channel reflection map and a 1-channel brightness map.

2. The method of claim 1, wherein, The feature extraction network comprises: a convolutional layer for extracting low-level features from the 3-channel reflectance map; a swin transformer encoder, which is used for extracting a semantic feature from the 3-channel reflection map or the normal light image and outputting to an upernet decoder; an upernet decoder, which is used for preliminarily decoding the semantic feature to obtain a 1 / 4 resolution feature map of the original image.

3. The method of claim 1, wherein, The low-light enhancement module comprises: in succession a first deconvolutional layer, a first activation layer, a second deconvolutional layer, a second activation layer, for up-sampling the semantic feature map; a splicing layer, which is used for splicing the low-level feature and the upsampled semantic feature to obtain a spliced feature map; In series Convolutional layers and Sigmoid, used to decode the spliced feature map to obtain an enhanced 3-channel reflectance map and a 1-channel brightness map, the 3-channel reflectance map serving as a low-illumination enhancement result.

4. The method of claim 1, wherein, The semantic segmentation module sequentially comprises a first deconvolutional layer, a first activation layer, a second deconvolutional layer, a second activation layer, used for decoding the 1 / 4 resolution feature map to obtain a semantic segmentation result map containing the category of each pixel point.

5. The method according to any one of claims 1 to 4, characterized in that, the total loss function calculation formula during training is as follows: wherein, represents the image enhancement loss of the light intensity enhancement network, represents the image enhancement loss of the low-light enhancement module, represents the image semantic segmentation loss of the semantic segmentation module, respectively represent the reconstruction loss, the reflection loss, and the illumination component smoothness loss of the light intensity enhancement network, respectively represent the reconstruction loss, the reflection loss, and the illumination component smoothness loss of the low-light enhancement module, represents the weight parameter, represents the number of categories, represents the category judgment of a variable, 1 if the categories are the same, and 0 if the categories are different, represents the predicted probability that the observation sample belongs to the category .

6. The method of claim 1, wherein, The training process uses a normal light semantic segmentation dataset, a low-light semantic segmentation dataset and a low-light image enhancement dataset together for training, and the specific mode comprises: In each round of iteration during the training process, there are fixed 3 input pictures in a batch, which are 3 pictures from cityscapes, nightcity and darkface respectively, representing the normal light semantic segmentation dataset, the low-light semantic segmentation dataset and the low-light image enhancement dataset respectively; The cityscapes dataset data only needs to obtain a semantic segmentation result map after passing through the network without low-light enhancement, and the semantic segmentation result map is used to calculate the semantic segmentation task loss; The nightcity dataset data passes through the network to obtain a semantic segmentation result image and a low-illumination enhancement result image at the same time, the semantic segmentation result image is used to calculate a semantic segmentation task loss, and the low-illumination enhancement result image is used to calculate a low-illumination enhancement loss. The darkface dataset data passes through the network to obtain a low-illumination enhancement result, which is used to calculate a low-illumination enhancement loss.

7. A system suitable for all-day image low-light enhancement and semantic segmentation joint system, characterized in that, Comprise: a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, so that the method in any one of claims 1 to 6 is executed.

Images