A dark and weak light semantic segmentation method and device

Abstract

The application belongs to the field of image processing, and provides a dark and weak light semantic segmentation method and device, which comprises the following steps: adopting an image processing model, inputting source domain data pictures I S , reference domain data pictures I R and target domain data pictures I T ; performing image enhancement on the source domain data pictures I S through a composite image enhancement module to obtain enhanced domain data pictures I E ; inputting I S , I R , I E , I T , and processing through the image processing model; obtaining good I T pseudo labels through the image processing model for self-training; stripping the encoder and the decoder from the image processing model; and directly outputting I T prediction; the application further reduces dataset differences through the composite image enhancement module, the image processing model and the self-training module.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically a method and apparatus for semantic segmentation in low light conditions. Background Technology

[0002] Semantic segmentation is a fundamental task in computer vision, aiming to assign an independent semantic label to each pixel in an image to aid in various downstream visual tasks. However, while existing semantic segmentation models have achieved considerable success, most are designed for well-lit daytime scenes. When applied to low-light environments, their performance often drops drastically, which is disastrous for safety-critical tasks such as autonomous driving. Furthermore, in low-light conditions, certain regions become difficult to identify, even for humans. This significantly increases the cost of labeling, making the re-collection and labeling of data adapted for low-light conditions tedious, redundant, and time-consuming.

[0003] To address the aforementioned issues, domain-adaptive semantic segmentation methods based on transfer learning have been proposed to improve the segmentation performance of low-light scene images (target domain data) by leveraging knowledge learned from well-lit daytime scene images (source domain data). However, due to the poor lighting conditions and significant noise in low-light scenes, directly transferring knowledge learned from daytime scenes to low-light scenes is not ideal. Therefore, existing methods typically use a daytime scene dataset with a coarse correspondence to the target domain as a reference domain to guide predictions in the target domain. Since both are daytime scenes, knowledge transfer from the source domain data to the reference domain is relatively effective; therefore, the key is how to utilize the reference domain to guide predictions in the target domain. Thus, a method is needed to more effectively utilize the coarse correspondence between the reference and target domain data to facilitate knowledge transfer to the target domain and improve the model's segmentation performance for low-light scene images.

[0004] The structural diagram of the existing technology is as follows Figure 13 and Figure 14 These represent the training and inference phases of existing methods, respectively.

[0005] Training phase: such as Figure 13 Specifically, firstly, the source domain data images The input is fed into the encoder module of the network to obtain the feature map. Then the features The input is fed into the decoder module to obtain the predicted probability. Prediction is obtained through Softmax. The prediction and source domain data tags By calculating cross-entropy loss And it propagates backwards to update the parameters of the encoder and decoder;

[0006] Furthermore, the target domain data images The data is sequentially input into the encoder and decoder to obtain the prediction. Then construct pseudo tags Used to calculate cross-entropy loss The specific construction process is as follows:

[0007] First, the target domain data image The data is sequentially input into the teacher encoder and decoder to obtain the predicted probabilities. The teacher encoder and decoder have the same network structure as the encoder and decoder, and their parameters are calculated from the encoder and decoder using a moving weighted average, as shown in the following formula.

[0008]

[0009] in, These are the parameters of the teacher encoder / decoder at time t+1. Here are the parameters of the teacher encoder / decoder at time t. The parameters of the encoder / decoder at time t. These are the weighting coefficients;

[0010] Secondly, the source domain data images Reference domain data image, target domain data image The data is sequentially input into the classroom encoder and decoder to obtain the predicted probability. ;

[0011] Furthermore, the target domain data images Reference domain data images and reference domain prediction probability The input is fed into an alignment network, which is pre-trained and based on... and Output the corresponding optical flow estimate, and predict the probability of the reference domain based on the optical flow estimate. Align to the target domain to obtain the alignment prediction. At the same time, the confidence probability of alignment is obtained. ;

[0012] Furthermore, based on this alignment prediction Alignment confidence probability and target domain prediction probability To obtain more optimized and improved predictions The specific formula is as follows:

[0013]

[0014] in, Indicates the weighting coefficient. The calculation formula is as follows:

[0015]

[0016] in, express The confidence score for the prediction is calculated using the following formula.

[0017]

[0018] in Represents information entropy. As a tuning hyperparameter;

[0019] The formula for determining whether a point belongs to a large static category is as follows.

[0020]

[0021] in The indices represent the height, width, and channel, respectively. Represents a large static collection of categories. , ;

[0022] Furthermore, for improving forecasting One-hot encoding is used to obtain pseudo-tags According to predictions and pseudo-tags Calculate loss Backpropagation is used to update the parameters;

[0023] Finally, by iterating the above process multiple times, once the model loss stops decreasing and the model parameters tend to be balanced, the trained network can be obtained.

[0024] Reasoning stage:

[0025] like Figure 14 For target domain data images By sequentially inputting these inputs into the trained encoder and decoder, the final semantic segmentation prediction can be output. .

[0026] The existing models described above perform well in predicting large static categories, but relatively poorly in predicting small static and dynamic categories. This is because the methods primarily utilize reference domain image data. and target domain data images The model has a coarse correspondence between positions and is aligned using optical flow. However, large static categories are easy to align, while small static and dynamic categories are difficult to align one by one. Furthermore, the model requires an additional large alignment network to be trained, which incurs significant training overhead.

[0027] To address the problems raised in the background art, those skilled in the art have proposed a method and apparatus for semantic segmentation in low-light conditions. Summary of the Invention

[0028] To address the aforementioned technical problems, this invention provides a method and apparatus for semantic segmentation in low light conditions, which solves the problem that existing models perform well in predicting large static categories, but perform relatively poorly in predicting small static and dynamic categories.

[0029] A semantic segmentation method for low-light conditions includes the following steps:

[0030] S201, employing an image processing model, inputs source domain data images. Reference domain data images and target domain data images ;

[0031] S202, using a composite image enhancement module to enhance the source domain data image Image enhancement is performed to obtain the enhanced domain data image. ;

[0032] S203, Input It is processed through an image processing model;

[0033] S204, obtained through image processing model, has good performance. The pseudo-labels are used for self-training. The image processing model includes a deformable attention reference module (DAR) and a prototype class alignment module (PCA).

[0034] S206 separates the encoder and decoder from the image processing model;

[0035] S207, direct output predict.

[0036] Preferably, step S203 includes the following steps:

[0037] S401, Input Image ;

[0038] S402, for ;

[0039] S403, features are first obtained by the encoder. ;

[0040] S404, Input decoder obtains prediction ;

[0041] S405, according to the label Calculate the loss;

[0042] S406, based on backpropagation of the loss, update the encoder and decoder parameters;

[0043] S407, for ;

[0044] S408, features are first obtained by the encoder. ;

[0045] S409, Using the DAR module to extract semantic reference features ;

[0046] S410, obtains improved features ;

[0047] S411, Decoder Decoding Features Get Prediction :Will The input is fed into the decoder to obtain the improved prediction probability. ,pass The prototype was obtained based on the moving weighted average. The specific calculation formula is as follows:

[0048]

[0049]

[0050] Where t represents time. The indices represent height, width, and category in that order. The moving weighted average coefficient, (for indicator functions);

[0051] according to and prototype The specific formula for calculating the feature distance loss is as follows:

[0052]

[0053]

[0054] in The indices represent height, width, and category in that order. One-hot encoding, This is the adjustment coefficient;

[0055] Get Prediction S412, according to the label Calculate the segmentation loss and the category feature alignment loss based on PCA;

[0056] S413, freeze encoder and decoder parameters, update DAR parameters;

[0057] S414, for ;

[0058] S415, features are first obtained by the encoder. ;

[0059] S416, Use the DAR module to extract semantic reference features ;

[0060] S417, Obtaining Improved Features ;

[0061] S418, Decoder Decoding Features Get Prediction ;

[0062] S419, calculate the category feature alignment loss based on PCA;

[0063] S420, freeze encoder and decoder parameters, update DAR parameters.

[0064] Preferably, the encoder module includes the following steps:

[0065] S501, for any input image ;

[0066] S502, first process the image ;

[0067] S503-505, passing through TransformerBlock1, and sequentially passing through... Second and Then through ;

[0068] S506, obtain the feature map ;

[0069] S507-509, passing through TransformerBlock2, successively passing through Second and Then through ;

[0070] S510, obtain the feature map ;

[0071] S511-513, passing through TransformerBlock3, and sequentially passing through... Second and Then through ;

[0072] S514, obtain the feature map ;

[0073] S515-517, passing through TransformerBlock4, and then sequentially passing through... Second and Then through ;

[0074] S518, obtain the feature map Finally, the encoder obtains the feature map. .

[0075] Preferably, the decoder module includes the following steps:

[0076] S601, for feature maps ;

[0077] S602, first align the feature map sizes to... ;

[0078] S603, then in the channel dimension merge;

[0079] S604 performs multi-scale dilated convolution;

[0080] S605 performs segmentation prediction using a segmentation head;

[0081] S606, obtaining the final prediction .

[0082] Preferably, the DAR module includes the following steps:

[0083] S701, for features ;

[0084] S702, firstly, obtains fusion features through channel merging and channel down-applying. ;

[0085] S703, by The offset is generated through convolution;

[0086] S704, by Sampling reference point;

[0087] S705, resampling features are obtained through bilinear interpolation based on offset and reference point. ;

[0088] S706, with for , for Perform attention query;

[0089] S708, using an MLP (Multilayer Perceptron);

[0090] S709, Repeat the above process 702-707. Finally, semantic references are obtained. .

[0091] Preferably, the PCA module includes the following steps:

[0092] S801, based on S411, obtains the corrected prediction probability. ;

[0093] S802, according to Calculate the prototypes for each category separately;

[0094] S803, for S412, calculate the corrected prediction probability. The loss is obtained by calculating the feature distance between the feature and the prototype of each category;

[0095] S804, for S419, calculate the corrected prediction probability. The loss is obtained by calculating the feature distance between the feature and the prototype of each category;

[0096] S805 uses the aforementioned loss to backpropagate, freezes the encoder and decoder, and updates the DAR parameters.

[0097] Preferably, the self-training module includes the following steps:

[0098] S901, for target domain data images ;

[0099] S902 utilizes encoder encoding features;

[0100] S903 utilizes decoder features;

[0101] S904, predicted ;

[0102] The image processing models in S905 and S203 obtain the teacher model based on the moving weighted average, and the predictions are obtained as shown in S414-418. , as a pseudo-tag;

[0103] S906, based on pseudo-labels and prediction Calculate the loss;

[0104] S907, backpropagation, updates encoder and decoder parameters.

[0105] A low-light semantic segmentation device includes a composite image enhancement module, a deformable attention reference module, a prototype category alignment module, and a self-training module.

[0106] Preferably, the composite image enhancement module generates enhancement domain data from source domain data through illumination enhancement and spatial enhancement, so that the source domain and enhancement domain form a source domain pair, which together with the target domain pair formed by the reference domain and the target domain form a new basic transfer unit.

[0107] The deformable attention reference module captures the semantic relevance of daytime and nighttime images within each domain pair by proposing a deformable attention Transformer, and improves the prediction of nighttime images based on this semantic relevance.

[0108] The prototype category alignment module transfers the ability of the deformable attention module to capture semantic relevance from source domain pairs to target domain pairs by utilizing prototype-based category feature alignment.

[0109] The self-training module uses the above-mentioned module and teacher network to obtain accurate predictions of the target domain image as pseudo-labels, and then uses self-training to retrain the encoder and decoder.

[0110] Compared with the prior art, the present invention has the following beneficial effects:

[0111] This invention utilizes a composite image enhancement module, employing both illumination and spatial enhancement, to construct enhanced domain data from source domain data. It also proposes, for the first time, a transfer method based on source domain pairs (source domain data and enhanced domain data) and target domain pairs (reference domain data and target domain data) as basic transfer units. Furthermore, it proposes a deformable attention semantic reference to capture semantic relevance within domain pairs, acting as a bridge for domain pair coupling. This allows low-light nighttime image features to reference daytime image features, improving their prediction capabilities. Finally, it proposes a category prototype-based alignment method to finely align features between different domain pairs, further reducing dataset discrepancies and improving computational accuracy. This effectively enhances the segmentation accuracy of low-light scenes. Attached Figure Description

[0112] Figure 1 This is a flowchart illustrating a low-light semantic segmentation method.

[0113] Figure 2 This is a flowchart illustrating the composite image enhancement module.

[0114] Figure 3 This is a flowchart illustrating the image processing model.

[0115] Figure 4 This is a flowchart of the encoder module;

[0116] Figure 5 This is a flowchart of the decoder module;

[0117] Figure 6 A flowchart illustrating the deformable attention reference module;

[0118] Figure 7 This is a flowchart illustrating the prototype-based class alignment module.

[0119] Figure 8 This is a flowchart illustrating the self-training module.

[0120] Figure 9 This is a schematic diagram illustrating one application scenario of the present invention;

[0121] Figure 10 This is a schematic diagram of the composite image enhancement module of the present invention;

[0122] Figure 11 This is a schematic diagram of the image processing model of the present invention;

[0123] Figure 12 This is a schematic diagram of the self-training module of the present invention;

[0124] Figure 13 This is a flowchart illustrating the existing technology.

[0125] Figure 14 This is a flowchart illustrating the existing technology. Detailed Implementation

[0126] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0127] Example 1: See Figure 9 It can capture images of the target domain in low light conditions. The data are sequentially input into the encoder and decoder to obtain an accurate segmentation prediction map, which can then be applied to downstream tasks.

[0128] See Figure 1 The method may include: determining the source domain data image. Reference domain data image Target domain data image .

[0129] Input source domain image The size can be expressed as .in, This indicates the number of channels in the image. The height of the image can be represented by the number of pixels in the vertical dimension. The width of the image can be represented by the number of pixels in the horizontal dimension.

[0130] Input source domain image The size can be This indicates that the image has three channels: red (Red, R), green (Green, G), and blue (Blue, B), with a height of 512 pixels and a width of 512 pixels.

[0131] Knowledge transfer can be achieved through the following feasible approach: First, enhance the source domain data images using a composite enhancement module. Obtain enhanced domain data image Then, through image processing models, knowledge is transferred from source domain pairs to target domain pairs. Simultaneously, within each domain pair, predictions for low-light scenes are corrected by referencing daytime scenes. This achieves good image processing model performance. Pseudo-labels are used for self-training; during the inference phase, the encoder and decoder are extracted from the image processing model for prediction, resulting in good target domain low-light images. The prediction.

[0132] Combination Figure 10 Further explanation of the composite enhancement module:

[0133] Source domain data images Illumination enhancement was performed using the CycleGAN model, resulting in an enhanced image with a size of [size missing]. ;

[0134] The illuminated image is then spatially enhanced using homography, thin-platespline (TPS), and affine-TPS to obtain the enhanced domain image. The size is ;

[0135] Combination Figure 11 The image processing model will be further explained below:

[0136] For source domain data images Enhanced domain data images For example, if a batch of images contains 2 images, then the scale is 2. First, input the encoder to obtain feature maps. , The dimensions are respectively , , ;

[0137] Will Input to the decoder to obtain the decoded prediction. All scales 19 represents the number of categories;

[0138] According to the label All sizes Calculate the cross-entropy loss, backpropagate, and update the encoder and decoder parameters;

[0139] For source domain data images Enhanced domain data images All scales First, input the encoder to obtain... ;

[0140] Will In Input is fed into the DAR module to obtain semantic reference features. , The dimensions are as follows: ;

[0141] Will Add to corresponding To obtain improved features ;

[0142] Will The input is fed into the decoder to obtain the improved prediction probability. The size is ;

[0143] According to the label The size is Calculate the cross-entropy loss, backpropagate and update the DAR parameters. At this time, the encoder and decoder parameters are frozen and not updated.

[0144] pass The prototype was obtained based on the moving weighted average. The size is The specific calculation formula is as follows:

[0145]

[0146]

[0147] Where t represents time. The indices represent height, width, and category in that order. The moving weighted average coefficient, For indicator functions;

[0148] according to and prototype The specific formula for calculating the feature distance loss is as follows:

[0149]

[0150]

[0151] in The indices represent height, width, and category in that order. One-hot encoding, This is the adjustment coefficient;

[0152] According to the loss The DAR parameters are backpropagated and updated. At this time, the encoder and decoder parameters are frozen and not updated.

[0153] For reference domain data images Target domain data image All scales First, input the encoder to obtain... ;

[0154] Will In Input is fed into the DAR module to obtain semantic reference features. , The dimensions are as follows: ;

[0155] Will Add to corresponding To obtain improved features ;

[0156] Will The input is fed into the decoder to obtain the improved prediction probability. The size is ;

[0157] according to and prototype The specific formula for calculating the feature distance loss is as follows:

[0158]

[0159]

[0160] According to the loss The DAR parameters are backpropagated and updated. At this time, the encoder and decoder parameters are frozen and not updated.

[0161] Combination Figure 12 Further explanation of the self-training module:

[0162] For reference domain data images Target domain data image All scales First, input the teacher encoder to obtain... The teacher encoder has the same structure as the encoder, and its parameters are calculated by moving weighted average.

[0163] Will In Input into the teacher's DAR to obtain semantic reference features , The dimensions are as follows: The teacher's DAR module has the same structure as the DAR module, and the parameters are calculated by moving weighted average.

[0164] Will Add to corresponding To obtain improved features ;

[0165] Will The input is fed into the teacher decoder to obtain the improved prediction probability. The size is The teacher decoder has the same structure as the decoder, and its parameters are calculated using a moving weighted average.

[0166] Based on the improved prediction probability Obtaining pseudo tags The size is ;

[0167] For target domain data images , scale is Input encoder to obtain ;

[0168] Will Input to decoder to obtain prediction The size is ;

[0169] according to and Calculate the cross-entropy loss, backpropagate, and update the encoder and decoder parameters.

[0170] by For example, combined with Figure 6 Further explanation of the DAR module:

[0171] First of all and (This is) ), all sizes First, after embedding, the size becomes ;

[0172] Will and Merging at the channel layer yields... The size is Then, after 3x3 convolution, feature fusion and downsampling are performed to obtain... The size is ;

[0173] right First, an offset is generated by depthwise separable convolution. The size is ,in middle This represents 4 offsets, 2 represents the batch size, the second 2 represents the number of channels, channel 0 represents the x-coordinate, and channel 1 represents the y-coordinate;

[0174] right Sampling points are obtained through sampling. The size is ;

[0175] By bilinear interpolation, based on and Obtaining resampled features The size is

[0176] right Embedding is performed using a 3x3 convolution as the query. We use 3x3 convolutions for embedding as keys and values, perform cross-domain attention queries, and output the weighted result. The size is ;

[0177] Will Input to ;

[0178] Repeat the above process ,in It can be flexibly adjusted according to the actual situation to obtain the final result. middle The size is

[0179] First of all and (This is) ), all sizes First, after embedding, the size becomes ;

[0180] Will and Merging at the channel layer yields... The size is Then, after 3x3 convolution, feature fusion and downsampling are performed to obtain... The size is ;

[0181] right First, an offset is generated by depthwise separable convolution. The size is ,in middle This represents 4 offsets, 2 represents the batch size, the second 2 represents the number of channels, channel 0 represents the x-coordinate, and channel 1 represents the y-coordinate;

[0182] right Sampling points are obtained through sampling. The size is ;

[0183] By bilinear interpolation, based on and Obtaining resampled features The size is

[0184] right Embedding is performed using a 3x3 convolution as the query. We use 3x3 convolutions for embedding as keys and values, perform cross-domain attention queries, and output the weighted result. The size is ;

[0185] Will Input to ;

[0186] Repeat the above process ,in It can be flexibly adjusted according to the actual situation to obtain the final result. middle The size is

[0187] The reasoning stage will be explained further below:

[0188] When the loss functions no longer decrease and tend to stabilize, stop model training and test a single target domain image. , size is The code passes through the encoder and decoder sequentially to obtain the final prediction. Size is .

[0189] A low-light semantic segmentation device includes a composite image enhancement module, a deformable attention reference module, a prototype category alignment module, and a self-training module.

[0190] The composite image enhancement module generates enhancement domain data from source domain data through illumination enhancement and spatial enhancement, so that the source domain and enhancement domain form a source domain pair, which in turn forms a new basic transfer unit with the target domain pair formed by the reference domain and the target domain. Illumination enhancement transforms the source domain data from daytime to low-light nighttime data, while spatial enhancement simulates the viewpoint offset relationship between the reference domain and the target domain. Through illumination and spatial enhancement, the consistency of illumination and spatial differences between the source domain pair and the target domain pair is ensured.

[0191] The Deformable Attention Reference Module captures the semantic relevance of daytime and nighttime images within each domain pair by proposing a Deformable Attention Transformer, and improves the prediction of nighttime images based on this semantic relevance. The Deformable Attention Reference Module first learns the ability to capture semantic relevance based on labels in the source domain pair, and then applies the module to the target domain pair to capture the semantic relevance within the target domain pair and improve the prediction of the target domain image.

[0192] The prototype category alignment module transfers the ability of the deformable attention module to capture semantic relevance from source domain pairs to target domain pairs by utilizing prototype-based category feature alignment. For features in the source domain pair that are improved by semantic relevance, relevant features are extracted for each category based on the label and the prototype is calculated. Then, the feature distance between the improved features and the prototype of the target domain pair is calculated. The loss function is used to bring the distance between prototypes of the same category closer and the distance between prototypes of different categories farther, thereby further aligning features at the category level and promoting domain-invariant semantic relevance.

[0193] The self-training module uses the aforementioned modules and the teacher network to obtain accurate predictions of the target domain image as pseudo-labels. It then retrains the encoder and decoder using self-training, thus eliminating the need for additional variable attention and relevance modules during the inference phase and achieving the same inference speed as the baseline.

[0194] The method is more accurate and can effectively improve the segmentation accuracy of low light scenes. In the public dataset, compared with the existing technology proposed in the background technology, the method has a 1.9% higher mIoU on the DarkZurich validation set and a 1.7% higher mIoU on the test set. It is the best in both static and dynamic categories. In addition, the model has stronger generalization and robustness. For the generalization test, the model has a 6.3% higher mIoU than the proposed patent on the BDD100k-night test set [1] without retraining, and a 1% higher mIoU on the Nighttime Driving test set.

[0195] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A semantic segmentation method for low-light conditions, characterized in that, Includes the following steps: S201, employing an image processing model, inputs source domain data images. Reference domain data images and target domain data images ; S202, using a composite image enhancement module to enhance the source domain data image Image enhancement is performed to obtain the enhanced domain data image. ; S203, Input It is processed through an image processing model; S204, obtained through image processing model, has good performance. The pseudo-labels are self-trained. The image processing model includes a deformable attention reference module (DAR) and a prototype class alignment module (PCA). The DAR module is used to capture intra-domain semantic reference features, and the PCA module is used to achieve class feature alignment. S206 separates the encoder and decoder from the image processing model; S207, direct output predict; S203 includes the following steps: S401, Input Image ; S402, for ; S403, features are first obtained by the encoder. ; S404, Input decoder obtains prediction ; S405, according to the label Calculate the loss; S406, based on backpropagation of the loss, update the encoder and decoder parameters; S407, for ; S408, features are first obtained by the encoder. ; S409, Using the DAR module to extract semantic reference features ; S410, obtains improved features ; S411, Decoder Decoding Features Get Prediction : Will The input is fed into the decoder to obtain the improved prediction probability. ,pass The prototype was obtained based on the moving weighted average. The specific calculation formula is as follows: ； ； Where t represents time. The indices represent height, width, and category in that order. The moving weighted average coefficient, For indicator functions; according to and prototype The specific formula for calculating the feature distance loss is as follows: ； ； in The indices represent height, width, and category in that order. One-hot encoding, This is the adjustment coefficient; Get Prediction ; S412, according to the label Calculate the segmentation loss and the category feature alignment loss based on PCA; S413, freeze encoder and decoder parameters, update DAR parameters; S414, for ; S415, features are first obtained by the encoder. ; S416, Use the DAR module to extract semantic reference features ; S417, Obtaining Improved Features ; S418, Decoder Decoding Features Get Prediction ; S419, calculate the category feature alignment loss based on PCA; S420: Freeze encoder and decoder parameters, update DAR parameters.

2. The low-light semantic segmentation method as described in claim 1, characterized in that: The encoder includes the following steps: S501, for any input image ; S502, first process the image ; S503-505, passing through TransformerBlock1, and then sequentially passing through... Second and Then through ; S506, obtain the feature map ; S507-509, passing through TransformerBlock2, successively passing through Second and Then through ; S510, obtain the feature map ; S511-513, passing through TransformerBlock3, successively passing through Second and Then through ; S514, obtain the feature map ; S515-517, passing through TransformerBlock4, and then sequentially passing through... Second and Then through ; S518, obtain the feature map Finally, the encoder obtains the feature map. .

3. The low-light semantic segmentation method as described in claim 1, characterized in that: The decoder includes the following steps: S601, for feature maps ; S602, first align the feature map sizes to... ; S603, then in the channel dimension merge; S604 performs multi-scale dilated convolution; S605 performs segmentation prediction using a segmentation head; S606, obtaining the final prediction .

4. The low-light semantic segmentation method as described in claim 1, characterized in that: The DAR module includes the following steps: S701, for features ; S702 first obtains fused features through channel merging and channel downsampling. ; S703, by The offset is generated through convolution; S704, by Sampling reference point; S705, resampling features are obtained through bilinear interpolation based on offset and reference point. ; S706, with for , for Perform attention query; S708, using an MLP (Multilayer Perceptron); S709, Repeat the above process 702-707. Finally, semantic references are obtained. .

5. The low-light semantic segmentation method as described in claim 1, characterized in that: The PCA module includes the following steps: S801, based on S411, obtains the corrected prediction probability. ; S802, according to Calculate the prototypes for each category separately; S803, for S412, calculate the corrected prediction probability. The loss is obtained by calculating the feature distance between the feature and the prototype of each category; S804, for S419, calculate the corrected prediction probability. The loss is obtained by calculating the feature distance between the feature and the prototype of each category; S805 uses the aforementioned loss to backpropagate, freezes the encoder and decoder, and updates the DAR parameters.

6. The low-light semantic segmentation method as described in claim 1, characterized in that: The self-training process includes the following steps: S901, for target domain data images ; S902 utilizes encoder encoding features; S903 utilizes decoder features; S904, predicted ; The image processing models in S905 and S203 obtain the teacher model based on the moving weighted average, and the predictions are obtained as shown in S414-418. , as a pseudo-tag; S906, based on pseudo-labels and prediction Calculate the loss; S907, backpropagation, updates encoder and decoder parameters.

7. A low-light semantic segmentation device, used in conjunction with any of the low-light semantic segmentation methods described in claims 1-6, characterized in that: It includes a composite image enhancement module, a deformable attention reference module, a prototype category alignment module, and a self-training module.

8. The low-light semantic segmentation device as described in claim 7, characterized in that: The composite image enhancement module generates enhancement domain data from source domain data through illumination enhancement and spatial enhancement, so that the source domain and enhancement domain form a source domain pair, which together with the target domain pair formed by the reference domain and the target domain form a new basic transfer unit. The deformable attention reference module captures the semantic relevance of daytime and nighttime images within each domain pair by proposing a deformable attention Transformer, and improves the prediction of nighttime images based on this semantic relevance. The DAR module transfers the ability of the PCA module to capture semantic relevance from source domain pairs to target domain pairs by utilizing prototype-based category feature alignment. The self-training module uses the above-mentioned module and teacher network to obtain accurate predictions of the target domain image as pseudo-labels, and then uses self-training to retrain the encoder and decoder.

Images