Night semantic segmentation method and system based on passive multi-level collaborative distillation

Abstract

The invention belongs to the technical field of unmanned driving environment perception, and provides a night semantic segmentation method and system based on passive multi-level collaborative distillation, and the technical scheme is as follows: firstly, initializing a teacher network and a student network based on pre-training model parameters under a normal illumination condition; then, teacher network parameters are fixed, and a teacher network is updated according to student network parameters in a momentum smooth propagation mode; then, based on a prediction result of a pre-training source model on the night image, selecting first K pixel points with the highest confidence coefficient to generate a pseudo tag; and finally, inputting the normal illumination image and the night image into a teacher network and a student network respectively, and realizing multi-level knowledge migration through structure perception level alignment, semantic consistency constraint optimization and frequency domain collaborative fusion. No night annotation data is needed, source domain original data does not need to be accessed, and the data cost and the privacy risk are remarkably reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of unmanned environment perception technology, and particularly relates to a night semantic segmentation method and system based on passive multi-level collaborative distillation. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Semantic segmentation is a core technology for environment perception of autonomous driving and mobile robots, and its task is to assign semantic class labels to each pixel in an image. However, in real open road scenes, night low-light conditions seriously affect image quality, causing problems such as insufficient image brightness, significant noise, blurred details, and color distortion, which leads to obvious performance degradation of semantic segmentation models trained under normal lighting conditions.

[0004] In the prior art, the common solution is to collect and label a large number of night image data, or to use unsupervised domain adaptation methods to realize transfer from normal lighting domain (source domain) to night domain (target domain). However, the former is costly and difficult to label; the latter does not require target domain labeling, but still requires access to source domain data and labeling throughout, which has data storage burden and privacy disclosure risk, especially when sensitive geographic and identity information are involved, which limits the feasibility of actual deployment. SUMMARY

[0005] To solve at least one of the technical problems in the background art, the present application provides a night semantic segmentation method and system based on passive multi-level collaborative distillation, which realizes robust adaptation of the model from normal lighting to night environment only by constructing a multi-level cognitive distillation mechanism through any normal weather and night image pair, without using any night labeled data and without touching the original normal lighting training data, only relying on the pre-trained model parameters.

[0006] To achieve the above purpose, the present application adopts the following technical solutions: The first aspect of the present application provides a night semantic segmentation method based on passive multi-level collaborative distillation, comprising the following steps: Based on the pre-trained semantic segmentation model parameters under normal lighting conditions, the teacher network and the student network are initialized respectively, and the teacher network is updated based on the student network; Based on the pre-trained semantic segmentation model, the night image is inferred to obtain a semantic segmentation result, the confidence distribution of each semantic class is calculated based on the semantic segmentation result, and the pseudo label is generated according to the confidence distribution; The semantic segmentation model is trained based on a normal illumination image, a night image and a pseudo label, including: inputting the normal illumination image into a teacher network to extract a teacher feature map, inputting the night image into a student network to extract a student feature map, combining a confidence distribution to generate the pseudo label, the teacher feature map and the student feature map, introducing structure perception hierarchical alignment, semantic consistency constraint optimization and frequency domain collaborative fusion to train the semantic segmentation model through multi-level collaborative distillation. The trained semantic segmentation model is used to infer a to-be-processed night image to obtain a final semantic segmentation result.

[0007] Further, when the teacher network is updated based on the student network, the teacher network parameters are fixed, and the teacher network parameters are updated through momentum smoothing propagation based on the student network parameters.

[0008] Further, the confidence distribution of each semantic category is calculated based on the semantic segmentation result, and the pseudo label is generated according to the confidence distribution, including: For each semantic category, the confidence distribution is independently calculated, and a dynamic Top-K selection strategy is used to independently select K pixel points with the highest confidence for each semantic category to generate the pseudo label.

[0009] Further, the pseudo label is represented as: , wherein, represents a semantic category value of the generated pseudo label map at a pixel position , is a prediction probability output of a source model, represents a pixel value of the night image at a pixel coordinate , represents a conditional probability, which represents a confidence or probability of the source model predicting that the pixel belongs to a semantic category under the condition that the visual information of the night image at the pixel position (h, w) is given, represents that no valid semantic label is assigned, represents a set of K pixel points with the highest confidence in the category c.

[0010] Further, the semantic segmentation model is trained through multi-level collaborative distillation by combining the pseudo label, the teacher feature map and the student feature map, introducing structure perception hierarchical alignment, semantic consistency constraint optimization and frequency domain collaborative fusion, including: The salient regions in the night image and the normal illumination image are extracted and aligned respectively to generate an attention mask, and the structure consistency of the student and the teacher network in the feature space is constrained by combining the attention mask. The pseudo-labels are used as supervision signals, and the extracted teacher feature maps and student feature maps are mapped to the embedding space. In the embedding space, a dynamic prototype vector is constructed for each category, and semantic consistency constraints of intra-class and inter-class relationships are constructed by combining the prototype vectors of each category. The teacher feature map and the student feature map are respectively subjected to Fast Fourier Transform to obtain the frequency domain representation. The frequency domain signal is decomposed into amplitude and phase. The alignment of high and low frequency features is constrained by amplitude consistency loss and phase consistency loss.

[0011] Furthermore, when constraining the structural consistency of the student and teacher networks in the feature space using attention masks, the consistency loss function is expressed as follows: , in, Represents the consistency loss function. and These represent the student and teacher networks respectively. l Feature mapping of layers, and These represent nighttime images and images under normal lighting conditions, respectively. Chamfer distance is used to measure the structural consistency between two point sets. This represents a weighting coefficient used to balance the contributions of different network layers.

[0012] Furthermore, in the embedding space, a dynamic prototype vector is constructed for each category, represented as: , , , in, This represents an attention weight vector. Let represent the embedding vector of the teacher feature map at position (h,w). Let represent the embedding vector of the teacher feature map after weighting at position (h,w). The teacher model is represented as a semantic category. The calculated dynamic prototype vector, This represents the teacher feature map at position (h,w). It is a set of pixels representing the category determined by the pseudo-label. Indicates the pixel position The pseudo-label value generated at that location, This represents element-wise multiplication. Represents a scalar count used for normalization.

[0013] The second aspect of the present application provides a night semantic segmentation system based on passive multi-level collaborative distillation, comprising: A model initialization module configured to initialize a teacher network and a student network respectively based on pre-trained semantic segmentation model parameters under normal lighting conditions, and update the teacher network based on the student network; A pseudo-label generation module configured to perform inference on a night image based on the pre-trained semantic segmentation model to obtain a semantic segmentation result, calculate a confidence distribution of each semantic class based on the semantic segmentation result, and generate a pseudo-label according to the confidence distribution; A multi-level collaboration module configured to train the semantic segmentation model based on normal lighting images, night images and pseudo-labels, including inputting the normal lighting images into the teacher network to extract teacher feature maps, inputting the night images into the student network to extract student feature maps, generating the pseudo-labels in combination with the confidence distribution, the teacher feature maps and the student feature maps, introducing structure perception level alignment, semantic consistency constraint optimization and frequency domain collaborative fusion for multi-level collaborative distillation to train the semantic segmentation model, and obtaining a trained semantic segmentation model; A semantic segmentation module configured to perform inference on a to-be-processed night image based on the trained semantic segmentation model to obtain a final semantic segmentation result.

[0014] The third aspect of the present application provides a computer readable storage medium.

[0015] A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the night semantic segmentation method based on passive multi-level collaborative distillation as described above.

[0016] The fourth aspect of the present application provides a computer device.

[0017] A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the steps of the night semantic segmentation method based on passive multi-level collaborative distillation as described above.

[0018] Compared with the prior art, the present application has the following advantages: The present application does not require any night annotation data and does not require access to source domain original data, significantly reducing data cost and privacy risk; through multi-level collaborative learning from spatial structure to frequency domain features, the discriminative ability of the model under low light conditions is enhanced; the frequency domain decomposition and high-low frequency separation distillation mechanism is introduced to improve the adaptability of the model to night image noise and blur, and the present application has good practicability and generalizability.

[0019] Advantages of the additional aspects of the application will become apparent in the following description, which is given for the purpose of illustration and is not meant to limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which constitute a part of this specification, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification. The embodiments of the application, together with their

[0021] Figure 1 is a night semantic segmentation method flow chart based on passive multi-level collaborative distillation provided by an embodiment of the application; Figure 2 is network initialization and teacher network updating provided by an embodiment of the application; Figure 3 is pseudo-label generation provided by an embodiment of the application; Figure 4 is a multi-level knowledge migration flow chart provided by an embodiment of the application. DETAILED DESCRIPTION

[0022] The application will be further described below with reference to the drawings and embodiments.

[0023] It should be noted that the following detailed description is illustrative only and is not intended to limit the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0024] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, they refer to the presence of a feature, step, operation, device, component and / or combinations thereof.

[0025] Since most semantic segmentation models currently have a significant decline in performance under night low light conditions, and lack large-scale high-quality night annotation data, the present application designs a multi-level collaborative distillation learning framework based on the principle of "using only pre-trained model parameters without source domain data". This method realizes effective knowledge transfer from normal light domain to night domain through teacher-student network structure, combined with structure perception alignment, semantic consistency constraint and frequency domain collaborative fusion.

[0026] Embodiment one As shown in Figure 1 The present embodiment provides a night semantic segmentation method based on passive multi-level collaborative distillation, comprising the following steps: Step 1: Based on the parameters of the semantic segmentation model pre-trained under normal lighting conditions, initialize the teacher network and student network respectively, and update the teacher network based on the student network; In this embodiment, the pre-trained semantic segmentation model can be such as DeepLabV2 or Segformer; In this embodiment, the teacher network and student network have the same structure, and the initial weights are derived from a pre-trained semantic segmentation model under normal lighting conditions. like Figure 2 As shown, when updating the teacher network based on the student network, the teacher network parameters are fixed, and the teacher network parameters are updated by using the momentum smoothing propagation method based on the student network parameters. The teacher network parameters do not participate in gradient backpropagation and are only used as the target network during the student network training process. The update formula is as follows: , in, and They represent the first n The parameters of the teacher network and student network after the next iteration. Indicates the first n The parameters of the teacher network after -1 iteration. The momentum coefficient is set to 0.999 in this embodiment to maintain the stability of the teacher network parameters.

[0027] Step 2: Perform reasoning on the nighttime image based on the pre-trained semantic segmentation model to obtain semantic segmentation results. Calculate the confidence distribution of each semantic category based on the semantic segmentation results and generate pseudo-labels based on the confidence distribution. like Figure 3 As shown, a source domain pre-trained model is used for nighttime images. Perform reasoning for each semantic category c The confidence distribution is calculated independently, and a dynamic Top-K selection strategy is adopted to independently select the K pixels with the highest confidence for each semantic category to generate pseudo-labels. , is represented as: , in, This indicates the pixel position of the generated pseudo-label image. The semantic category value on This is the predicted probability output of the source model. Representing nighttime images In pixel coordinates Pixel value at that location, The representation is a conditional probability, representing the visual information at pixel location (h, w) in a given nighttime image. Under the condition of source model predict the confidence (or probability) that the pixel belongs to a semantic class , indicates that no valid semantic label is assigned to it and it is completely ignored in the subsequent training process, indicates the set of K pixels with the highest confidence in class c, and in this example K = 20% of the total pixels in class c.

[0028] Step 3: input the normal illumination image into the teacher network to extract the teacher feature map, input the night image into the student network to extract the student feature map, combine the confidence distribution to generate pseudo labels, teacher feature map and student feature map, introduce structure perception level alignment, semantic consistency constraint optimization and frequency domain collaborative fusion for multi-level collaborative distillation to train the semantic segmentation model, and obtain the trained semantic segmentation model. As shown in Figure 4 , it specifically includes the following steps: Step 301, respectively extract the salient regions in the night image and the normal illumination image and align them to generate an attention mask, and combine the attention mask to constrain the structural consistency of the student and teacher networks in the feature space; As an implementation, the pre-trained salient region detection network (PoolNet) is used to extract the salient regions in the night image and the normal image, and align the salient regions in the night image and the normal image, so as to realize the alignment of important regions in the night image and the normal image. As an implementation, channel attention and spatial attention mechanisms are used to decompose and enhance features to generate an attention mask .

[0029] As an implementation, the Chamfer distance is used to constrain the structural consistency of the student and teacher networks in the feature space, and the consistency loss function is as follows: , wherein, and represent the feature mapping of the first l layer of the student and teacher networks respectively, and represent the night image and the normal illumination image respectively, is the Chamfer distance, which is used to measure the structural consistency between two point sets, is a weight coefficient used to balance the contribution of different network layers.

[0030] Step 302: Input the normal lighting image into the teacher network to extract the teacher feature map, and input the nighttime image into the student network to extract the student feature map. Map the teacher feature map and the student feature map into the embedding space. In the embedding space, construct a dynamic prototype vector for each category. Specifically, the steps include the following: Step 3021: To prevent the student model from deviating from the correct semantic space during the adaptation process, high-confidence pseudo-labels are first used. This provides direct supervision signals, and the loss function ensures that the student model's base predictions remain consistent with the teacher model's authoritative predictions: , in, Losses due to pseudo-supervision The total number of valid pixels for pseudo-tags For indicator functions, Let be the student network's predicted probability for class c at position (h,w); Step 3022: Input the normal lighting image into the teacher network to extract the teacher feature map, and input the nighttime image into the student network to extract the student feature map. Map the teacher feature map and the student feature map into the embedding space. In this embodiment, a learnable projection head is introduced. The projection head consists of two fully connected layers and an activation function, designed to map the features of the teacher and student models into a low-dimensional, highly discriminative embedding space. , , in, and Let represent the embedding vectors of the teacher's feature map and the student's feature map at position (h, w), respectively. and These represent the teacher's projector head and the student's projector head, respectively. and Let represent the teacher feature map and student feature map at position (h, w), respectively. and These represent the first-layer weights of the projection heads for the teacher network and the student network, respectively. and These represent the second-layer weights of the projection heads for the teacher network and the student network, respectively. It is the ReLU activation function. and These represent the first-layer bias of the projector heads for the teacher network and the student network, respectively. and The second layer of the projection head of the teacher network and the student network, respectively.

[0031] In particular, and , and and and , and These parameters are the learnable parameters of the projection head, and the teacher and student projection heads do not share parameters, which provides greater flexibility for the model to learn feature representations in respective domains.

[0032] Step 3023, constructing a dynamic prototype vector for each class in the embedding space; To avoid noise and highlight important features, a channel attention mechanism is introduced to weight the prototype: , , , where, is an attention weight vector, is the weighted embedding vector, which better represents the essential features of the class, is the dynamic prototype vector calculated by the teacher model for the semantic class , is the class pixel set determined by the pseudo label, represents the pseudo label value generated at pixel position , represents element-wise multiplication, is a scalar count for normalization.

[0033] Step 3024, constructing intra-class and inter-class relationship semantic consistency constraints combining the prototype vector of each class; In this embodiment, a loss is designed to capture the intra-class and inter-class relationship, to reduce the distance between the student embedding and the teacher prototype, and to push away the prototypes of different classes. This is the core of the semantic consistency constraint:

[0034] where, is the cosine similarity, is a temperature hyperparameter, is the dynamic prototype vector calculated by the student model for the semantic class , is a general iterative variable for all classes; Step 303, the teacher feature map and the student feature map are respectively subjected to fast Fourier transform to obtain a frequency domain representation, the frequency domain signal is decomposed into amplitude and phase, and the alignment of high and low frequency features is constrained by amplitude consistency loss and phase consistency loss: In this embodiment, the intermediate layer features of the teacher network and the student network are respectively subjected to fast Fourier transform (FFT) to obtain a frequency domain representation . .

[0035] The frequency domain signal is decomposed into amplitude A and phase P two parts: , The FFT formula is as follows:

[0036] Wherein, is the pixel value in the spatial domain located at , is the complex value in the frequency domain located at .

[0037] The alignment of high and low frequency features is constrained by amplitude consistency loss and phase consistency loss : , , Wherein, and represent the amplitude and phase obtained by decomposing the frequency domain signal of the student network, and represent the amplitude and phase obtained by decomposing the frequency domain signal of the teacher network; The adaptive filter is applied to the high frequency component to retain important detail information, and the attention mechanism is used to enhance the global structure information of the low frequency component.

[0038] The total loss of the frequency domain cooperative fusion is: , Wherein, and are learnable weight coefficients.

[0039] Finally, the total training loss of the student network is the weighted sum of the above loss functions: , Wherein, , are the weight hyperparameters for balancing each loss, which are both set to 1 in this example. The student network parameters are updated by minimizing The teacher network parameters are updated by momentum smoothing.

[0040] Step 4: performing inference on the to-be-processed night image based on the trained semantic segmentation model to obtain a final semantic segmentation result.

[0041] To illustrate the effectiveness of the method, Cityscapes, a large-scale urban street scene dataset for semantic understanding, is used as the source dataset. The dataset contains 5,000 high-quality pixel-level labeled images collected in 50 different urban driving scenarios, including 2,975 training images, 500 validation images, and 1,525 test images according to the standard division, covering 19 semantic categories. For the target domain dataset: the night subset in ACDC (400 / 102 / 500 images in the training set / validation set / test set) provides paired images under night conditions and their corresponding normal weather versions.

[0042] The results of the method on the ACDC night subset are shown in Table 1, and the method of the application achieves better results.

[0043] Table 1 Comparison of segmentation results on the ACDC night subset

[0044] The experimental results show that the method significantly improves the segmentation accuracy and robustness of the model under night conditions without any night labeled data, and has strong adaptability to noise and blur.

[0045] Embodiment Two The embodiment provides The model initialization module is configured to initialize the teacher network and the student network based on the pre-trained semantic segmentation model parameters under normal lighting conditions, and update the teacher network based on the student network; The pseudo-label generation module is configured to perform inference on the night image based on the pre-trained semantic segmentation model to obtain a semantic segmentation result, calculate the confidence distribution of each semantic category based on the semantic segmentation result, and generate a pseudo-label according to the confidence distribution; The multi-level collaborative module is configured to train the semantic segmentation model based on the normal lighting image, the night image, and the pseudo-label, including: inputting the normal lighting image into the teacher network to extract a teacher feature map, inputting the night image into the student network to extract a student feature map, combining the confidence distribution, the pseudo-label, the teacher feature map, and the student feature map, introducing structure perception level alignment, semantic consistency constraint optimization, and frequency domain collaborative fusion for multi-level collaborative distillation to train the semantic segmentation model, and obtaining a trained semantic segmentation model. The semantic segmentation module is configured to perform inference on the to-be-processed night image based on the trained semantic segmentation model to obtain a final semantic segmentation result.

[0046] It should be noted that the specific implementation of the embodiment of the application is similar to the specific implementation of the embodiment of the application. For details, please refer to the description of the method part. In order to reduce redundancy, this part is not described here.

[0047] Embodiment three The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps in the embodiment of the application.

[0048] Embodiment four The embodiment provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the embodiment of the application are realized.

[0049] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system, or a computer program product. Therefore, the application can be in the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer usable program code.

[0050] The application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one or more flows and / or blocks. Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0051] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0052] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0053] Those of ordinary skill in the art can understand that all or part of the flow of the above-mentioned embodiment method can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the flow of the above-mentioned embodiment method. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.

[0054] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those of ordinary skill in the art can make various changes and modifications to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A night semantic segmentation method based on passive multi-level collaborative distillation, characterized in that, The method comprises the following steps: Based on the pre-trained semantic segmentation model parameters under normal lighting conditions, the teacher network and the student network are initialized respectively, and the teacher network is updated based on the student network; Based on the pre-trained semantic segmentation model, the night image is inferred to obtain a semantic segmentation result, the confidence distribution of each semantic class is calculated based on the semantic segmentation result, and the pseudo label is generated according to the confidence distribution; Based on the normal lighting image, the night image and the pseudo label, the semantic segmentation model is trained, including: the normal lighting image is input into the teacher network to extract the teacher feature map, the night image is input into the student network to extract the student feature map, the pseudo label is generated by combining the confidence distribution, the teacher feature map and the student feature map, the structure perception hierarchical alignment, the semantic consistency constraint optimization and the frequency domain collaborative fusion are introduced to perform multi-level collaborative distillation on the semantic segmentation model to train the semantic segmentation model; Based on the trained semantic segmentation model, the night image to be processed is inferred to obtain the final semantic segmentation result.

2. The method of claim 1, wherein the method is a passive multi-level collaborative distillation based night-time semantic segmentation method. When the teacher network is updated based on the student network, the teacher network parameters are fixed, and the teacher network parameters are updated by momentum smoothing propagation based on the student network parameters.

3. The method of claim 1, wherein the method is a passive multi-level collaborative distillation based night-time semantic segmentation method. The confidence distribution of each semantic class is calculated based on the semantic segmentation result, and the pseudo label is generated according to the confidence distribution, including: For each semantic class, the confidence distribution is calculated independently, and the dynamic Top-K selection strategy is adopted to independently select K pixel points with the highest confidence for each semantic class to generate a pseudo label.

4. The method of claim 1, wherein the method is a passive multi-level collaborative distillation based night-time semantic segmentation method. The pseudo label is represented as: ， wherein, denotes the semantic class value of the generated pseudo label map at pixel position , is the prediction probability output of the source model, denotes the night image the pixel value at pixel coordinate , denotes a conditional probability that represents the confidence or probability that the source model predicts that the pixel belongs to the semantic class given the visual information of the night image at pixel position (h, w), , denotes that no valid semantic label is assigned to it, denotes the set of K pixel points with the highest confidence in class c.

5. The method of claim 1, wherein the method is a passive multi-level collaborative distillation based night-time semantic segmentation method. The pseudo label, the teacher feature map and the student feature map are combined, the structure perception hierarchical alignment, the semantic consistency constraint optimization and the frequency domain collaborative fusion are introduced to perform multi-level collaborative distillation on the semantic segmentation model to train the semantic segmentation model, including: The significant areas in the night image and the normal lighting image are extracted and aligned respectively to generate an attention mask, and the attention mask is combined to constrain the structural consistency of the student and the teacher network in the feature space; The pseudo label is used as a supervision signal, and the teacher feature map and the student feature map are mapped to an embedding space, a dynamic prototype vector is constructed for each class in the embedding space, and a class-intra-class relationship semantic consistency constraint is constructed by combining the prototype vector of each class; The teacher feature map and the student feature map are respectively subjected to fast Fourier transform to obtain frequency domain representation, the frequency domain signal is decomposed into amplitude and phase, and the alignment of high and low frequency features is constrained by amplitude consistency loss and phase consistency loss.

6. The method of claim 5, wherein the method is based on passive multi-level synergistic distillation of night-time semantics. When the attention mask is combined to constrain the structural consistency of the student and the teacher network in the feature space, the structural consistency of the student and the teacher network in the feature space is constrained by Chamfer distance, and the consistency loss function is represented as: ， where, denotes the consistency loss function, and denotes the feature map of the l layer of the student and teacher network respectively, and denotes the night image and normal illumination image respectively, is the Chamfer distance, which is used to measure the structural consistency between two point sets, denotes a weight coefficient for balancing the contribution of different network layers.

7. The method of claim 5, wherein the method is based on passive multi-level synergistic distillation of night-time semantics. In the embedding space, a dynamic prototype vector is constructed for each class, and is represented as: ， ， ， wherein, denotes an attention weight vector, denotes an embedding vector of the teacher feature map at position (h, w), denotes an embedding vector of the weighted teacher feature map at position (h, w), denotes the semantic class predicted by the teacher model the computed dynamic prototype vector, denotes the teacher feature map at position (h, w), is the set of class pixels determined by the pseudo labels, denotes the pseudo label value generated at pixel position (h, w), denotes an element-wise multiplication, denotes a scalar count for normalization.

8. A night semantic segmentation system based on passive multi-level collaborative distillation, characterized in that, It comprises: The model initialization module is configured to initialize the teacher network and the student network based on the pre-trained semantic segmentation model parameters under normal lighting conditions, and to update the teacher network based on the student network. The pseudo-label generation module is configured to: perform inference on the night image based on the pre-trained semantic segmentation model to obtain a semantic segmentation result, calculate a confidence distribution of each semantic category based on the semantic segmentation result, and generate a pseudo-label according to the confidence distribution; The multi-level collaborative module is configured to: train the semantic segmentation model based on the normal illumination image, the night image, and the pseudo-label, including: inputting the normal illumination image into a teacher network to extract a teacher feature map, inputting the night image into a student network to extract a student feature map, combining the confidence distribution, the teacher feature map, and the student feature map to generate the pseudo-label, introducing structure perception level alignment, semantic consistency constraint optimization, and frequency domain collaborative fusion to perform multi-level collaborative distillation on the semantic segmentation model, and training the semantic segmentation model to obtain a trained semantic segmentation model; The semantic segmentation module is configured to: perform inference on a to-be-processed night image based on the trained semantic segmentation model to obtain a final semantic segmentation result.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the night semantic segmentation method based on passive multi-level collaborative distillation according to any one of claims 1-7.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the night semantic segmentation method based on passive multi-level collaborative distillation according to any one of claims 1-7.

Images