A low-light scene end-to-end semantic segmentation method

Abstract

The application discloses a low-light scene end-to-end semantic segmentation method, which belongs to the field of robot perception and computer vision. The application proposes a hierarchical gating convolution unit, which can expand the deep feature receptive field of the network and restore the edge texture. In order to solve the problem of low contrast of texture features between objects, the application proposes a double closed loop dichotomy matching algorithm to establish a total loss norm considering unsupervised illumination enhancement loss and supervised union loss, so as to minimize the two losses by the Hungarian algorithm. The method can realize low-light scene end-to-end semantic segmentation. The long-tail distribution of common and sparse class samples in the training data leads to the feature learning deviation of the network to common classes and rare classes. In this regard, the application designs a rare class sampling strategy to increase the sampling probability of rare classes during the training process, so as to obtain a model with excellent performance.

Description

Technical Field

[0001] This invention, based on deep learning technology, studies an end-to-end semantic segmentation network for low-light or nighttime scenes. First, it proposes a hierarchical gated convolutional unit module, which not only expands the network's receptive domain to recover edge textures but also enhances the contrast of texture features between objects. Then, by employing a dual-loop binary matching algorithm, a total loss paradigm that comprehensively considers unsupervised illumination enhancement loss and supervised intersection-union ratio (IoU) loss is constructed, allowing these two losses to be jointly minimized using the Hungarian algorithm. This achieves a training method for an end-to-end semantic segmentation network particularly suitable for low-light scene processing. To address the long-tail distribution problem of common and sparse class samples in the training data, this invention designs a rare class sampling strategy to increase the sampling frequency of rare classes during training, thereby training a model with superior performance. This invention belongs to the field of computer vision, specifically involving deep learning and pattern recognition technologies. Background Technology

[0002] Semantic segmentation, a fundamental task in computer vision, aims to label each pixel of a given image as an object category and plays a crucial role in many applications, such as robot perception, autonomous driving, and medical imaging. While deep learning techniques have made significant progress in semantic segmentation of well-lit daytime images, their performance is significantly hampered in low-light scenes. This problem becomes even more challenging due to the lack of large-scale, fine-grained labeled datasets for semantic segmentation in low-light scenes. This is primarily attributed to the variability of illumination changes, which cannot be reliably captured with limited image sets, and the substantial time and labor required to label large datasets.

[0003] Significant progress has been made in semantic segmentation for low-light scenes. Some studies have proposed illumination enhancement models to preprocess dark images to resemble those under natural daylight before training the semantic segmentation model. Other studies have designed guided illumination adaptation frameworks that utilize stylized synthetic images and unlabeled real images to leverage scene correspondences across time. However, these progressive adaptation methods typically require training multiple semantic segmentation models independently, and the performance of semantic segmentation heavily depends on the model trained during the preprocessing stage. Summary of the Invention

[0004] In the field of low-light image segmentation, there are two main challenges: 1) limited visibility of targets, and 2) reduced color contrast of edge textures between different targets, both of which hinder the segmentation of target edges. To our knowledge, there is currently no end-to-end semantic segmentation network designed for low-light conditions. The main challenge stems from the inability of existing methods to simultaneously minimize unsupervised illumination enhancement loss and supervised cross-union ratio (CUI) loss, due to potential conflicts between the minimization objectives of these two losses. The former is based on the illumination histogram curve, while the latter is based on the cross-entropy of pixel classification. To address this issue, this invention proposes a dual-loop binary matching algorithm, which establishes a comprehensive loss criterion. This invention also proposes a hierarchical gated convolutional unit. It not only expands the deep feature receptive field of the original network while maintaining almost no change in the number of parameters, but also utilizes the powerful edge feature extraction capability of gated convolution to alleviate the problem of edge information loss in low-light images. Therefore, this invention proposes an end-to-end semantic segmentation network for low-light scenes, realizing an end-to-end training method that enables the network to simultaneously perform illumination enhancement and semantic segmentation on low-light images.

[0005] Step 1: Training dataset preparation. First, three classic city street scenes, Cityscapes, Dark_Zurich, and Nighttime Driving, are used to construct input sample points: a) The image size is adjusted to 1152×768 or 768×1152; b) Random data augmentation is applied to the dataset.

[0006] Step 2: For an image X∈R W*H*3 R is a real number, the first and second dimensions represent the length and width of the feature map, and the third dimension represents the number of channels in the feature map. To transform an image X∈R... W*H*3 The input to the feature extraction module based on a deep residual structure first needs to convert the image information into a one-dimensional sequence information that conforms to the input of the backbone network. Preprocessing includes image patch partitioning and merging, which solves the input problem. Feature extraction first passes through a 7×7 2D convolutional layer and a 3×3 pooling layer, then through multiple residual modules and cascaded pyramid pooling modules. Each residual module differs from the ResNet network's residual module only in one aspect: replacing the ordinary 3×3 2D convolutional layer with a novel 3×3 2D convolutional layer with an internally hierarchical residual structure. This novel convolution first evenly divides the input feature map into s feature map subsets, which are then divided by x... i This means that, where i∈{1,2,…,s}, each feature map subset has the same spatial size as the input feature map, but the number of channels becomes 1 / s, and each x except x1 has a different spatial size. i Both operations undergo one 2D convolution operation with a 3×3 kernel and one 2D gated convolution operation. These two convolution operations are represented by K... i() indicates that y is used to represent y i K represents i The output feature map of (), the final output of this convolutional structure is The addition here means stitching the resulting feature maps together in the third channel, y i The calculation formula is: The pyramid pooling module consists of three pooling channels with pooling sizes of 12×12, 6×6, and 3×3. Each pooling layer is followed by a 2D convolutional layer with a 1×1 kernel to transform the third dimension of the feature map to 256. Ultimately, this step can extract image feature maps of three different scales, i.e., X1∈R. 69*69*256 X2∈R 35*35*256 X3∈R 18*18*256 ; Set X3∈R 18 *18*256 X4∈R is obtained by a single 2D convolutional layer with a 3×3 kernel. 9*9*256 Then, through a 2D convolutional layer with a 3×3 kernel, X5∈R is obtained. 5*5*256 The input is an image X∈R. W*H*3 The output is X1∈R 69*69*256 The network portion is defined as an inner loop, which is used for illumination enhancement preprocessing; while the input is an image X∈R. W*H*3 The output is the semantic segmentation result X∈R W*H*1 The semantic segmentation network is defined as the outer ring;

[0007] Step 3: To enhance image illumination, we adopt the loss function designed in Zero DCE++: in The loss consists of exposure control loss, color constancy loss, and illumination smoothness loss; the same weights as in Zero DCE++ are used, i.e., ω1 = 1, ω2 = 0.5, ω3 = 20; the proposed network generates two fixed-size prediction arrays: one for the light-enhanced image and the other for the semantic mask; to leverage end-to-end training, the label assignment problem is defined as a bipartite graph matching problem, and then the Hungarian algorithm is used to find the minimum value of the bipartite graph as the loss value L. h : Where α is a hyperparameter balancing the effects of classification and segmentation, set to 0.8; c k With c i Let represent the category labels of the k-th and i-th truth objects, respectively, and Indicates that the i-th prediction belongs to category c k The probability of m; i and t k These are the masks for the i-th prediction and the k-th truth object, respectively. N is the number of matching pairs, and Pσ is the mask for the i-th prediction and the k-th truth object, respectively. (i) (c i C(i,k) is the predicted probability of the current class, and C(i,k) is the pairwise matching score based on DICE. i ,t k ) is the pixel-level binary cross-entropy loss;

[0008] Step 4: Since the optimization objectives of the two loss functions are inconsistent, creating the total loss as a weighted sum of the two loss functions will lead to training non-convergence. To address this, a dual-loop bipartite graph matching training strategy is proposed: 1) The inner loop is responsible for unsupervised training, thereby enhancing image illumination; the loss L1 of this loop training reflects the effect of the shallow network of the encoder on image illumination enhancement. 2) The outer loop is responsible for supervised training to improve semantic segmentation accuracy; the L2 loss of this loop training indicates the overall accuracy of end-to-end network semantic segmentation. The inner loop loss value L1 is added to the outer loop for training; the inner loop loss is multiplied by the backpropagation weight β1, and the backpropagation process is represented as L(); in the forward propagation of the outer loop, it is represented as F(); during loss calculation, the loss is further multiplied by β2 for backpropagation; this is added to the loss of the inner loop training, and this loss is weighted by β2, resulting in: Where L1(x) k L2(x) k F(L2(x)) represents the loss obtained by the inner loop and outer loop networks after training on the k-th sample x, respectively. k-1 )+β1L1(x k-1 )) indicates that when training the (k-1)th sample x, the inner and outer loops are updated with the loss values ​​L2 and β1L1 respectively, and then the network is forward propagated again to obtain L2 and L1 respectively, which are used as the training loss values ​​for the kth sample x. and These correspond to the initial values ​​for L2 and L1, respectively; during the initial training phase, the network is trained once using a set of initial random samples x0 to obtain the initial loss L. 0 L2 is the next forward propagation... The initial values ​​of β2 and β3 are set to 1, and the values ​​of β1 and β3 are selected from 0.01, 0.1, 1, 10, and 100 to minimize the loss. This dual-loop training method ensures gradient descent in the network for both unsupervised training to enhance image illumination and supervised training for semantic segmentation accuracy, thereby guaranteeing the convergence of the end-to-end training of the entire network.

[0009] Step 5: Propose a sparse class sampling optimization strategy to alleviate the long-tail distribution problem of sparse classes and common classes in the training samples; the sampling frequency f for each class c. c ∈(0,1], calculated based on the number of pixels with class c∈[1,C]: Where N is a one-hot label y used for training. (i,j,c) The number of samples; the sampling probability P(c)∈(0,1) of a certain class c is defined as its frequency f. c Functions: Where T∈(0,1] modulates the smoothness of the distribution; a higher T leads to a more uniform distribution, while a lower T focuses on rarer classes, thus increasing their selection probability. This mechanism ensures that classes with lower frequencies have a higher probability of being sampled; f c’ It is a certain type of sampling frequency, and f c The same definition is used to calculate the number of pixel samples for all categories; during the sampling process of each source instance, a class c is selected according to the probability distribution P(c); taking into account rare classes (composed of low f) c (This indicates that) it is often associated with several common classes (by high f) c (Indicates) coexistence. This strategy is advantageous for sampling rare classes more frequently than common classes, in order to achieve a fairer class distribution in the resampled dataset.

[0010] Step 6: After training, the image is input into the feature extraction module based on the deep residual structure. The resulting feature maps are processed by spatial pooling pyramids to obtain multiple sets of fixed-size feature maps, among which the shallow feature map X1∈R 69*69*256 The feature map X7∈R is obtained by concatenating it with the feature map from the spatial feature recovery module. 69*69*280 The original image X∈R is input into a prototype generation module that generates a set of prototypes of the same size as the input image. The spatial feature recovery module consists of three cascaded 3×3 gated convolutional layers, 3×3 2D convolutional layers, and a bilinear interpolation layer, with the original image X∈R... W*H*3 As input, gated convolutional layers are used to extract edge features from the image. Transposed convolutional layers and bilinear interpolation layers are used to change the number of channels, ultimately generating a feature map X6∈R.69*69*24 splice it to X1∈R 69*69*256 From this, we obtain X7∈R 69*69*280 The prototype generation module based on multi-layer transposed convolution upsampling consists of four transposed convolutional layers with 3×3 kernels and one 2D convolutional layer with 1×1 kernel, yielding P∈R. W*H*32 Deep feature map X2∈R 35*35*256 X3∈R 18 *18*256 X3∈R 18*18*256 X4∈R 9*9*256 X5∈R 5*5*256 A set of coefficient matrices is obtained through the coefficient matrix prediction module based on mask position regression. This module first passes the input through two cascaded 3×3 2D convolutional layers, then through three parallel 3×3 2D convolutional layers, resulting in three sets of coefficient matrices, which are the class confidence prediction matrices C1∈R. W*H*(81*a) Bounding box prediction matrix C2∈R W*H*(4*a) With mask prediction matrix C3∈R W*H*(32*a) Where 'a' represents the number of preset anchor frames. Within certain limits, a larger 'a' results in more accurate position prediction, but the number of parameters increases exponentially, slowing down the algorithm's execution speed. Experiments have shown that a value of 3–8 is optimal. Let C1∈R W*H*(81*a) C2∈R W*H*(4*a) C3∈R W*H*(32*a) Each with P∈R W*H*32 After performing matrix multiplication, the matrix is ​​then processed by the Sigmoid activation function, the expression of which is as follows: Where x is the input value and e is the base of the natural logarithm; the prototype is trimmed and spliced ​​to obtain the final instance segmentation result. This process can be described as follows: M = Sigmoid(PC1) T +PC2 T +PC3 T ) Where P is a W×H×k prototype mask matrix, W and H are the width and height of the input image to the segmentation network, k is the number of prototype groups, and C1 T C2 T C3 T Let C1, C2, and C3 be the transpose matrices respectively, M∈R W*H*N To generate the final mask, the third channel N is normalized using Batch Normalization to obtain the semantic segmentation result Y∈R. W*H*1Experiments demonstrate that the proposed network, employing a dual-loop bipartite graph matching training strategy, achieves mIoU (mean average Intersection of Union) of 81.2%, 57.9%, and 56.9% on the Cityscapes, DarkZurich, and Nighttime Driving public datasets, respectively, after end-to-end training on the training samples. Furthermore, the inference speed on a single NVIDIA 3090Ti graphics card is 22.3 FPS (frames per second). Therefore, transferring this method to low-light environment semantic segmentation tasks is feasible and has significant application value for achieving efficient, accurate, and real-time semantic segmentation in low-light scenes. Attached Figure Description

[0011] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0012] Figure 1 This is a schematic diagram of the end-to-end semantic segmentation network in this invention.

[0013] Figure 2 This is a schematic diagram comparing a regular convolution module with the gated residual convolution module of this invention.

[0014] Figure 3 This is a schematic diagram of the feature extraction network in this invention.

[0015] Figure 4 This is a visualization comparing the semantic segmentation network of this invention with existing methods on the NighttimeDriving public dataset.

[0016] Figure 5 This is a visualization comparing the test results of the semantic segmentation network of this invention on a self-made dataset with images captured and labeled by the built-in camera of a quadruped robot, and existing methods. Detailed Implementation

[0017] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific examples. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0018] This invention proposes an end-to-end semantic segmentation network for low-light scenes, realizing an end-to-end training method that enables the network to simultaneously perform illumination enhancement and semantic segmentation on low-light images.

[0019] The method proposed in this invention includes the following five parts: (1) dataset preparation; (2) sparse class sampling optimization strategy; (3) overall network structure; (4) inter-layer cascaded gated residual convolution module; and (5) double-loop bipartite graph matching training strategy.

[0020] (1) Dataset preparation: First, three classic city street scene public semantic segmentation Cityscapes, Dark_Zurich and Nighttime Driving are used to construct input sample points: a) The image size is adjusted to 1152×768 or 768×1152; b) Random data augmentation is applied to the dataset; c) The training set, validation set and test set are divided according to the format provided by the public dataset.

[0021] (2) Sparse Class Sampling Optimization Strategy: A sparse class sampling optimization strategy is proposed to alleviate the long-tail distribution problem of sparse classes and common classes in the training samples; the sampling frequency f for each class c is... c ∈(0,1], calculated based on the number of pixels with class c∈[1,C]: Where N is a one-hot label y used for training. (i,j,c) The number of samples; the sampling probability P(c)∈(0,1) of a certain class c is defined as its frequency f. c Functions: Where T∈(0,1] modulates the smoothness of the distribution; a higher T leads to a more uniform distribution, while a lower T focuses on rarer classes, thus increasing their selection probability. This mechanism ensures that classes with lower frequencies have a higher probability of being sampled; f c’ It is a certain type of sampling frequency, and f c The same definition is used to calculate the number of pixel samples for all categories; during the sampling process of each source instance, a class c is selected according to the probability distribution P(c); taking into account rare classes (composed of low f) c (This indicates that) it is often associated with several common classes (by high f) c (Indicates) coexistence. This strategy is advantageous for sampling rare classes more frequently than common classes, in order to achieve a fairer class distribution in the resampled dataset.

[0022] (3) Inter-layer cascaded gated residual convolution module: This module first divides the input feature map into s uniform feature map subsets, and then... i This means that, where i∈{1,2,…,s}, each feature map subset has the same spatial size as the input feature map, but the number of channels becomes 1 / s, and each x except x1 has a different spatial size. iBoth operations undergo one 2D convolution operation with a 3×3 kernel and one 2D gated convolution operation. These two convolution operations are represented by K... i () indicates that y is used to represent y i K represents i The output feature map of (), the final output of this convolutional structure is The addition here means stitching the resulting feature maps together in the third channel, y i The calculation formula is: This inter-layer cascaded gated residual convolution module is used to construct the feature extraction network in the overall network architecture of this invention. A comparison diagram between it and a regular residual convolution module is shown below. Figure 2 As shown.

[0023] (4) Overall network structure: For a sample image X∈R in the training set W*H*3 R is a real number, the first and second dimensions represent the length and width of the feature map, and the third dimension represents the number of channels in the feature map. To transform an image X∈R... W*H*3 The input to the feature extraction module based on a deep residual structure first needs to convert the image information into a one-dimensional sequence that conforms to the input of the backbone network. Preprocessing includes image patch partitioning and merging, which addresses the input problem. Feature extraction first passes through a 2D convolutional layer with a 7×7 kernel and a pooling layer with a 3×3 size, then through multiple residual modules and cascaded pyramid pooling modules. Each residual module differs from the ResNet network's residual module only in that the 3×3 ordinary 2D convolutional layer is replaced with a cascaded gated residual convolutional module. The pyramid pooling module consists of three pooling channels with pooling sizes of 12×12, 6×6, and 3×3. Each pooling layer is followed by a 2D convolutional layer with a 1×1 kernel to increase the third dimension of the feature map to 256. Finally, this step extracts image feature maps of three different scales, i.e., X1∈R. 69*69*256 X2∈R 35*35*256 X3∈R 18*18*256 ; Set X3∈R 18*18*256 X4∈R is obtained by a single 2D convolutional layer with a 3×3 kernel. 9 *9*256 Then, through a 2D convolutional layer with a 3×3 kernel, X5∈R is obtained. 5*5*256 The feature extraction module, such as Figure 2 As shown; where the input is the image X∈R W*H*3 The output is X1∈R 69*69*256 The network portion is defined as an inner loop, which is used for illumination enhancement preprocessing; while the input is an image X∈R. W*H*3 The output is the semantic segmentation result Y∈R. W*H*1The overall network is defined as an outer loop; the image is input into a feature extraction module based on a deep residual structure, and the resulting feature maps are processed through a spatial pooling pyramid to obtain multiple sets of fixed-size feature maps, among which the shallow feature map X1∈R 69*69*256 The feature map X7∈R is obtained by concatenating it with the feature map from the spatial feature recovery module. 69*69*280 The original image X∈R is input into a prototype generation module that generates a set of prototypes of the same size as the input image. The spatial feature recovery module consists of three cascaded 3×3 gated convolutional layers, 3×3 2D convolutional layers, and a bilinear interpolation layer, with the original image X∈R... W*H*3 As input, gated convolutional layers are used to extract edge features from the image. Transposed convolutional layers and bilinear interpolation layers are used to change the number of channels, ultimately generating a feature map X6∈R. 69*69*24 splice it to X1∈R 69*69*256 From this, we obtain X7∈R 69*69*280 The prototype generation module based on multi-layer transposed convolution upsampling consists of four transposed convolutional layers with 3×3 kernels and one 2D convolutional layer with 1×1 kernel, yielding P∈R. W*H*32 Deep feature map X2∈R 35*35*256 X3∈R 18*18*256 X3∈R 18*18*256 X4∈R 9*9*256 X5∈R 5*5*256 A set of coefficient matrices is obtained through the coefficient matrix prediction module based on mask position regression. This module first passes the input through two cascaded 3×3 2D convolutional layers, then through three parallel 3×3 2D convolutional layers, resulting in three sets of coefficient matrices, which are the class confidence prediction matrices C1∈R. W *H*(81*a) Bounding box prediction matrix C2∈R W*H*(4*a) With mask prediction matrix C3∈R W*H*(32*a) Where 'a' represents the number of preset anchor frames. Within certain limits, a larger 'a' results in more accurate position prediction, but the number of parameters increases exponentially, slowing down the algorithm's execution speed. Experiments have shown that a value of 3–8 is optimal. Let C1∈R W*H*(81*a) C2∈R W*H*(4*a) C3∈R W*H*(32*a) Each with P∈R W*H*32 After performing matrix multiplication, the matrix is ​​then processed by the Sigmoid activation function, the expression of which is as follows: Where x is the input value and e is the base of the natural logarithm; the prototype is trimmed and spliced ​​to obtain the final instance segmentation result. This process can be described as follows: M = Sigmoid(PC1)T +PC2 T +PC3 T ) Where P is a W×H×k prototype mask matrix, W and H are the width and height of the input image to the segmentation network, k is the number of prototype groups, and C1 T C2 T C3 T Let C1, C2, and C3 be the transpose matrices respectively, M∈R W*H*N To generate the final mask, the third channel N is normalized using Batch Normalization to obtain the semantic segmentation result Y∈R. W*H*1 The semantic segmentation network of this invention is visualized by comparing its test results on the NighttimeDriving public dataset with existing methods, as shown below. Figure 4 As shown in the figure; the test results of the semantic segmentation network of the present invention on a self-made dataset of 1280×720 resolution images captured by the front Intel Realsence D455 camera of a quadruped robot and annotated are compared with existing methods. Figure 5 As shown.

[0024] (5) Dual-loop bipartite graph matching training strategy: This algorithm enables the network to perform end-to-end training for simultaneous illumination enhancement and semantic segmentation. First, to enhance image illumination, we adopt the loss function designed in Zero DCE++: in The loss consists of exposure control loss, color constancy loss, and illumination smoothness loss; the same weights as in Zero DCE++ are used, i.e., ω1 = 1, ω2 = 0.5, ω3 = 20; the proposed network generates two fixed-size prediction arrays: one for the light-enhanced image and the other for the semantic mask; to leverage end-to-end training, the label assignment problem is defined as a bipartite graph matching problem, and then the Hungarian algorithm is used to find the minimum of the bipartite graph as the loss value Lh. Where α is a hyperparameter balancing the effects of classification and segmentation, set to 0.8; c k With c i Let represent the category labels of the k-th and i-th truth objects, respectively, and Indicates that the i-th prediction belongs to category c k The probability of m; i and t k These are the masks for the i-th prediction and the k-th truth object, respectively. N is the number of matching pairs, and Pσ is the mask for the i-th prediction and the k-th truth object, respectively. (i)(c i C(i,k) is the predicted probability of the current class, and C(i,k) is the pairwise matching score based on DICE. i ,t k The first loss is the pixel-level binary cross-entropy loss. Secondly, since the optimization objectives of the two loss functions are inconsistent, creating the total loss as a weighted sum of the two loss functions leads to training non-convergence. To address this, a dual-loop bipartite graph matching training strategy is proposed: 1) The inner loop is responsible for unsupervised training, thereby enhancing image illumination; the loss L1 of this loop training reflects the effect of the shallow network of the encoder performing image illumination enhancement; 2) The outer loop is responsible for supervised training to improve semantic segmentation accuracy; the loss L2 of this loop training indicates the overall accuracy of end-to-end network semantic segmentation. The inner loop loss value L1 is added to the outer loop for training; the inner loop loss is multiplied by the backpropagation weight β1, and the backpropagation process is represented as L(); in the forward propagation of the outer loop, it is represented as F(); during loss calculation, the loss is further multiplied by β2 for backpropagation; this is added to the loss of the inner loop training, and this loss is weighted by β2, resulting in: Where L1(x) k L2(x) k F(L2(x)) represents the loss obtained by the inner loop and outer loop networks after training on the k-th sample x, respectively. k-1 )+β1L1(x k-1 )) indicates that when training the (k-1)th sample x, the inner and outer loops are updated with the loss values ​​L2 and β1L1 respectively, and then the network is forward propagated again to obtain L2 and L1 respectively, which are used as the training loss values ​​for the kth sample x. and These correspond to the initial values ​​for L2 and L1, respectively; during the initial training phase, the network is trained once using a set of initial random samples x0 to obtain the initial loss L. 0 L2 is the next forward propagation... The initial values ​​of β2 and β3 were set to 1, and the values ​​of β1 and β3 were selected from 0.01, 0.1, 1, 10, and 100 to minimize the loss. This dual-loop training method ensures gradient descent in the network for both unsupervised training to enhance image illumination and supervised training for semantic segmentation accuracy, thus guaranteeing the convergence of the end-to-end training of the entire network. All experiments were conducted on a desktop computer running Ubuntu, equipped with four NVIDIA RTX 3090 graphics cards, each with 24GB of RAM. The SGD optimizer was used, with a momentum setting of 0.9 and a weight decay of 5 × 10⁻⁶. -4For each model, we used the original image size of 2048×1024 as input, with an initial learning rate set to 10. -4 The momentum is fixed at 0.9. Furthermore, random cropping is applied, with a crop size of 512 and scaling ranging from 0.5 to 1.0, along with random horizontal flipping. The training batch size is set to 32, and the total number of iterations is 60,000. After training, a random image X∈R from the test set is used. W*H*3 Perform a forward inference process of the network to obtain the semantic segmentation result Y∈R W*H*1 .

[0025] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and not restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the equivalent meaning and scope of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0026] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An end-to-end semantic segmentation method for low-light scenes, characterized in that, Includes the following steps: Step 1: Training dataset preparation. First, three classic city street scenes, Cityscapes, Dark_Zurich, and Nighttime Driving, are used to construct input sample points: a) The image size is adjusted to 1152×768 or 768×1152; b) Random data augmentation is applied to the dataset. Step 2: For an image X∈R W*H*3 R is a real number, the first and second dimensions represent the length and width of the feature map, and the third dimension represents the number of channels in the feature map. To transform an image X∈R... W*H*3 The input to the feature extraction module based on deep residual structure first needs to convert the image information into one-dimensional sequence information. Preprocessing includes image patch partitioning and merging. Feature extraction first passes through a 7×7 2D convolutional layer and a 3×3 pooling layer, then through four sets of residual modules cascaded with pyramid pooling modules. Each residual module differs from the ResNet network's residual module only in one aspect: replacing the ordinary 3×3 2D convolutional layer with a novel 3×3 2D convolutional layer with an internally hierarchical residual structure. The input feature map is first evenly divided into s feature map subsets, x... i This means that, where i∈{1,2,…,s}, each feature map subset has the same spatial size as the input feature map, but the number of channels becomes 1 / s, and each x except x1 has a different spatial size. i Both operations undergo one 2D convolution operation with a 3×3 kernel and one 2D gated convolution operation. These two convolution operations are represented by K... i () indicates that y is used to represent y i K represents i The output feature map of (), the final output of this convolutional structure is The addition here means stitching the resulting feature maps together in the third channel, y i The calculation formula is: The pyramid pooling module consists of three pooling channels with pooling sizes of 12×12, 6×6, and 3×3. Each pooling layer is followed by a 2D convolutional layer with a 1×1 kernel to transform the third dimension of the feature map to 256. This step extracts image feature maps of three different scales, i.e., X1∈R. 69*69*256 X2∈R 35*35*256 X3∈R 18*18*256 ; Set X3∈R 18*18*256 X4∈R is obtained by a single 2D convolutional layer with a 3×3 kernel. 9*9*256 Then, through a 2D convolutional layer with a 3×3 kernel, X5∈R is obtained. 5 *5*256 The input is an image X∈R. W*H*3 The output is X1∈R 69*69*256 The network portion is defined as an inner loop, which is used for illumination enhancement preprocessing; while the input is an image X∈R. W*H*3 The output is the semantic segmentation result Y∈R. W*H*1 The overall semantic segmentation network is defined as the outer ring; Step 3: To enhance image illumination, a loss function designed in Zero DCE++ is used: in The loss consists of exposure control loss, color constancy loss, and illumination smoothness loss; the same weights as in Zero DCE++ are used, i.e., ω1 = 1, ω2 = 0.5, ω3 = 20; the proposed network generates two fixed-size prediction arrays: one for the light-enhanced image and the other for the semantic mask; to leverage end-to-end training, the label assignment problem is defined as a bipartite graph matching problem, and then the Hungarian algorithm is used to find the minimum value of the bipartite graph as the loss value L. h : Where α is a hyperparameter balancing the effects of classification and segmentation, set to 0.8; c k With c i Let represent the category labels of the k-th and i-th truth objects, respectively, and Indicates that the i-th prediction belongs to category c k The probability of m; i and t k These are the masks for the i-th prediction and the k-th truth object, respectively. N is the number of matching pairs, and P is the number of matching pairs. σ(i) (c i C(i,k) is the predicted probability of the current class, C(i,k) is the pairwise matching score based on DICE, and DICE(m) is the predicted probability of the current class. i ,t k ) is the pixel-level binary cross-entropy loss; Step 4: Propose a double-loop bipartite graph matching training strategy: 1) The inner loop is responsible for unsupervised training; the L1 loss of this loop reflects the effect of the shallow network of the encoder on image illumination enhancement; the inner loop is the network before the pyramid pooling module, and its input is X∈R. W*H*3 The output is X1∈R 69 *69*256 ; 2) The outer loop is responsible for supervised training; the L2 loss of this loop training indicates the accuracy of the overall end-to-end network semantic segmentation; the outer loop is the overall semantic segmentation network, and its input is X∈R. W*H*3 The output is Y∈R W*H*1 ; The inner loop loss value L1 is added to the outer loop for training; the inner loop loss is multiplied by the backpropagation weight β1, and the backpropagation process is represented as L(); in the forward propagation of the outer loop, it is represented as F(); during loss calculation, the loss is further multiplied by β2 for backpropagation; this is added to the loss of the inner loop training, and this loss is weighted by β2, resulting in: Where L1(x) k L2(x) k F(L2(x)) represents the loss obtained by the inner loop and outer loop networks after training on the k-th sample x, respectively. k-1 )+β1L1(x k-1 )) indicates that when training the (k-1)th sample x, the inner and outer loops are updated with the loss values ​​L2 and β1L1 respectively, and then the network is forward propagated again to obtain L2 and L1 respectively, which are used as the training loss values ​​for the kth sample x. and These correspond to the initial values ​​for L2 and L1, respectively; during the initial training phase, the network is trained once using a set of initial random samples x0 to obtain the initial loss L. 0 L2 is the next forward propagation... The initial values ​​of β2 and β3 are set to 1, and the values ​​of β1 and β3 are selected from 0.01, 0.1, 1, 10, and 100 to minimize the loss. This dual-loop training method ensures gradient descent in the network for both unsupervised training to enhance image illumination and supervised training for semantic segmentation accuracy, thereby guaranteeing the convergence of the end-to-end training of the entire network. Step 5: Propose a sparse class sampling optimization strategy to alleviate the long-tail distribution problem of sparse classes and common classes in the training samples; the sampling frequency f for each class c. c ∈(0,1], calculated based on the number of pixels with class c∈[1,C]: Where N is a one-hot label y used for training. (i,j,c) The number of samples; the sampling probability P(c)∈(0,1) of a certain class c is defined as its frequency f. c Functions: Where T∈(0,1] adjusts the smoothness of the distribution; f c’ It is a certain type of sampling frequency, and f c The same definition is used to calculate the number of pixel samples for all categories; during the sampling process of each source instance, a class c is selected according to the probability distribution P(c); Step 6: After training, the image is input into the feature extraction module based on the deep residual structure. The resulting feature maps are processed by spatial pooling pyramids to obtain multiple sets of fixed-size feature maps, among which the shallow feature map X1∈R 69*69*256 The feature map X7∈R is obtained by concatenating it with the feature map from the spatial feature recovery module. 69*69*280 The original image X∈R is input into a prototype generation module that generates a set of prototypes of the same size as the input image. The spatial feature recovery module consists of three gated convolutional layers with 3×3 kernels connected in series, a 2D convolutional layer with 3×3 kernels, and a bilinear interpolation layer. W*H*3 As input, gated convolutional layers are used to extract edge features from the image. Transposed convolutional layers and bilinear interpolation layers are used to change the number of channels, ultimately generating a feature map X6∈R. 69*69*24 splice it to X1∈R 69*69*256 From this, we obtain X7∈R 69 *69*280 The prototype generation module based on multi-layer transposed convolution upsampling consists of four transposed convolutional layers with 3×3 kernels and one 2D convolutional layer with 1×1 kernel, yielding P∈R. W*H*32 Deep feature map X2∈R 35*35*256 X3∈R 18*18*256 X3∈R 18*18*256 X4∈R 9*9*256 X5∈R 5*5*256 A set of coefficient matrices is obtained through the coefficient matrix prediction module based on mask position regression. This module first passes the input through two cascaded 3×3 2D convolutional layers, then through three parallel 3×3 2D convolutional layers to obtain three sets of coefficient matrices, which are the class confidence prediction matrices C1∈R. W*H*(81*a) Bounding box prediction matrix C2∈R W*H*(4*a) With mask prediction matrix C3∈R W *H*(32*a) Where a∈[3,8] represents the number of preset anchor frames; let C1∈R W*H*(81*a) C2∈R W*H*(4*a) C3∈R W*H*(32*a) Each with P∈R W*H*32 After performing matrix multiplication, the matrix is ​​then processed by the Sigmoid activation function, the expression of which is as follows: Where x is the input value and e is the base of the natural logarithm; the prototype is trimmed and spliced ​​to obtain the final instance segmentation result. This process can be described as follows: M=Digmoid(PC1 T +PC2 T +PC3 T ) Where P is a W×H×k prototype mask matrix, W and H are the width and height of the input image to the segmentation network, k is the number of prototype groups, and C1 T C2 T C3 T Let C1, C2, and C3 be the transpose matrices respectively, M∈R W*H*N To generate the final mask, the third channel N is normalized using Batch Normalization to obtain the semantic segmentation result Y∈R. W*H*1 .

Images