A passive online cross-domain semantic segmentation method, system, and terminal for rainy days

Abstract

This invention discloses a passive online cross-domain semantic segmentation method, system, and terminal for rainy weather. The method includes: determining a prior model and obtaining a feature prototype; inputting the target domain data stream into a teacher model and the prior model for prediction, obtaining a first normalization result, and obtaining a denoised pseudo-label based on the first normalization result; inputting the target domain data stream into a student model, obtaining a segmentation loss based on the student model's prediction result and the denoised pseudo-label, obtaining a second normalization result based on the student model's prediction result and the feature prototype, and obtaining a feature space structure consistency constraint based on the second normalization result and the first normalization result; supervising the model training based on the segmentation loss and the feature space structure consistency constraint to obtain a target student model; and inputting the data stream to be processed into the target student model to obtain the semantic classification result of the data stream to be processed. The model provided by this invention can achieve accurate semantic category segmentation in rainy weather.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a passive online cross-domain semantic segmentation method, system, terminal, and computer-readable storage medium for rainy days. Background Technology

[0002] Semantic segmentation is a key task in computer vision, with wide applications in autonomous driving, intelligent guidance for the visually impaired, and medical imaging diagnosis. Currently, semantic segmentation algorithms based on convolutional neural networks and Transformers have made significant progress under supervised learning paradigms. However, when faced with out-of-distribution scenes, the segmentation performance of these algorithms often deteriorates significantly due to domain differences.

[0003] However, most traditional semantic segmentation algorithms based on unsupervised domain adaptation suffer from two main problems. First, these methods typically require complete knowledge of both the source and target domains and necessitate training over multiple epochs during the domain adaptation process. Second, most of these methods assume a specific target domain and have limited long-term continuous domain adaptation capabilities. This makes them ill-suited for real-world scenarios such as rainy weather, where the complete information of the target domain is unknown, rainfall intensity varies continuously, data arrives in streaming form, and the source domain may be inaccessible due to privacy or confidentiality concerns.

[0004] Currently, online continuous domain semantic segmentation algorithms for rainy weather can be broadly categorized into self-training-based methods, normalization-based methods, and style transfer-based methods. Self-training-based methods are directly affected by the quality of pseudo-labels. Furthermore, during long-term domain adaptation, noise in the pseudo-labels accumulates, leading to a decline in pseudo-label quality and consequently impacting the algorithm's adaptive performance. Normalization-based methods struggle to handle situations with significant domain differences; that is, the model's performance improvement in the target domain after domain adaptation is limited. Style transfer-based methods transfer the target domain style to the source domain and then use the transferred source domain images and labels to train a student model, enabling it to adapt to the changing target domain. However, they are limited by the source domain data; when the source domain data becomes inaccessible, these methods fail.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide a passive online cross-domain semantic segmentation method, system, terminal, and computer-readable storage medium for rainy days. It aims to solve the problem that in existing online continuous domain adaptive semantic segmentation algorithms for rainy days, the noise in the self-trained pseudo-labels accumulates during long-term domain adaptation, causing a decline in pseudo-label quality and catastrophic forgetting, resulting in the inability to accurately segment semantic categories in rainy days.

[0007] To achieve the above objectives, this invention provides a passive online cross-domain semantic segmentation method for rainy days, comprising the following steps: Acquire the target domain data stream for rainy days and the source domain data stream for non-rainy days. Input the source domain data stream and the target domain data stream into the source pre-trained model for calculation to obtain the first target domain class feature prototype, the confidence of each batch of images in the source domain and the confidence of each batch of images in the target domain. Obtain the prior model based on the confidence of each batch of images in the target domain. The target domain data stream is input into the teacher model for feature extraction to obtain the first feature, and the target domain data stream is input into the prior model for prediction to obtain the prior prediction result. The first normalization result is obtained based on the first feature and the first target domain class feature prototype. The prior prediction result is denoised based on the first normalization result to obtain the denoised pseudo label. The target domain data stream is input into the student model to obtain the student model prediction result and the second feature. The segmentation loss is calculated based on the student model prediction result and the denoised pseudo-label. The second normalization result is obtained based on the second feature and the first target domain class feature prototype. The feature space structure consistency constraint is calculated based on the second normalization result and the first normalization result. The target loss function is calculated based on the segmentation loss and the feature space structure consistency constraint. The student model is trained by minimizing the target loss function to obtain the student model after one iteration. The parameters of the teacher model are updated by exponential moving average. The parameters of the student model after the first iteration are adjusted based on the confidence scores of each batch of images in the target domain and the confidence scores of each batch of images in the source domain. The adjusted student model is then used for the next iteration. After completing one round of iterations, the target student model is obtained. Obtain the data stream to be processed, input the data stream to be processed into the target student model for calculation, and obtain the semantic classification result of the data stream to be processed.

[0008] Optionally, the passive online cross-domain semantic segmentation method for rainy days, wherein the step of inputting the source domain data stream into a source pre-trained model for calculation, determining a prior model based on the calculation result, and obtaining a first target domain class feature prototype specifically includes: The source domain data stream is input into the source pre-trained model in batches. Feature extraction is performed, and the first power prototype of the source domain class features is calculated based on the extracted features. The quadratic prototype of source domain class features ; The target domain data stream is input into the source pre-trained model in batches. Calculations are performed to obtain the confidence score for each batch of images in the target domain. ,in, The ordinal number of the batch; Based on the confidence level of each batch of images in the target domain The prior model is obtained. : ; in, and To represent different confidence thresholds, The threshold for confidence changes when the domain distribution changes. Indicates the first Confidence level of the batch For the student model in the front domain; Based on the first power prototype of the source domain class characteristics and the quadratic prototype of the source domain class features The prototype of the first target domain class feature is obtained, wherein the prototype of the first target domain class feature includes the prototype of the first power of the first target domain class feature. and the quadratic prototype of the first target domain class features .

[0009] Optionally, in the passive online cross-domain semantic segmentation method for rainy days, the teacher model is defined as the student model. Exponential moving average; The step of obtaining the first normalization result based on the first feature and the first target domain class feature prototype specifically includes: Based on the first power prototype of the first target domain class features and the quadratic prototype of the first target domain class features Calculate the variance of the target domain category prototype. : According to the variance Computational Teacher Model The first feature extracted With category prototype The first normalized result of the distance between .

[0010] Optionally, the passive online cross-domain semantic segmentation method for rainy days, wherein the step of denoising the prior prediction result based on the first normalization result to obtain denoised pseudo-labels specifically includes: Use the first normalization result The prior prediction results generated by the prior model Denoising is performed to obtain denoised pseudo-labels. : ; in, This is a function that converts soft tags to one-hot encoded tags.

[0011] Optionally, the passive online cross-domain semantic segmentation method for rainy days, wherein the step of denoising the prior prediction result based on the first normalization result to obtain denoised pseudo-labels, further includes: Use the aforementioned noise-reducing pseudo-labels Calculate the prototype of the true first power class feature and the prototype of the real quadratic class features ; Based on the true first power class feature prototype and the real quadratic class feature prototype The first power prototype of the first target domain class features The prototype of the square of the features of the first target domain class The update is performed to obtain the first power prototype of the second target domain class feature. Prototype of the square of the second target domain class features : ; ; in, This is the prototype momentum coefficient.

[0012] Optionally, the passive online cross-domain semantic segmentation method for rainy days, wherein the step of calculating the segmentation loss based on the student model prediction result and the denoised pseudo-label, obtaining the second normalization result based on the second feature and the first target domain class feature prototype, and calculating the feature space structure consistency constraint based on the second normalization result and the first normalization result, specifically includes: Based on the student model prediction results and the denoised pseudo-labels Calculate the segmentation loss: ; in, For cross-entropy loss, and For hyperparameters; A second normalization result is obtained based on the second feature and the first target domain class feature prototype. The second normalization result... Compared with the first normalized result Between KL Minimizing the divergence yields the structural consistency constraint of the feature space. : ; in, KL The divergence between the normalization results, This represents the total number of categories in the current batch. c The ordinal number of the class. For the first c The first normalization result of the class feature prototype, For the first c The second normalization result of the class feature prototype.

[0013] Optionally, the passive online cross-domain semantic segmentation method for rainy days further includes: Starting from the second iteration, use the first power prototype of the second target domain class feature. The prototype of the square of the second target domain class features Calculate the first normalization result, and update the first power prototype of the second target domain class feature after each iteration. Prototype of the square of the second target domain class features .

[0014] Optionally, adjusting the student model parameters after the first iteration based on the confidence scores of each batch of images in the target domain and the source domain specifically includes: The initial parameter mixed momentum is calculated based on the confidence scores of each batch of images in the target domain and each batch of images in the source domain. Based on the distribution relationship between the target domain and the source domain, different attenuation coefficients are used to adjust the mixed momentum of the initial parameters. The target mixed momentum is obtained by attenuation. ; Based on the target mixed momentum The parameters of the student model after the first iteration are adjusted as follows: ; in, For the parameters of the adjusted student model, The parameters of the source pre-trained model, These are the parameters of the student model after the first iteration.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a passive online cross-domain semantic segmentation method system for rainy days, wherein the passive online cross-domain semantic segmentation method system for rainy days includes: The data preparation module is used to acquire the target domain data stream during rainy days and the source domain data stream during non-rainy days. The source domain data stream and the target domain data stream are respectively input into the source pre-trained model for calculation to obtain the first target domain class feature prototype, the confidence of each batch of images in the source domain and the confidence of each batch of images in the target domain. The prior model is obtained based on the confidence of each batch of images in the target domain. The pseudo-label denoising module is used to input the target domain data stream into the teacher model for feature extraction to obtain the first feature, and input the target domain data stream into the prior model for prediction to obtain the prior prediction result. Based on the first feature and the first target domain class feature prototype, a first normalization result is obtained. Based on the first normalization result, the prior prediction result is denoised to obtain a denoised pseudo-label. The loss calculation module is used to input the target domain data stream into the student model to obtain the student model prediction result and the second feature, calculate the segmentation loss based on the student model prediction result and the denoised pseudo-label, obtain the second normalization result based on the second feature and the first target domain class feature prototype, and calculate the feature space structure consistency constraint based on the second normalization result and the first normalization result. The model update module is used to calculate the target loss function based on the segmentation loss and the feature space structure consistency constraint, train the student model by minimizing the target loss function to obtain the student model after one iteration, and update the parameters of the teacher model by exponential moving average. The target model acquisition module is used to adjust the parameters of the student model after one iteration based on the confidence scores of each batch of images in the target domain and the confidence scores of each batch of images in the source domain, and then perform the next iteration on the adjusted student model to obtain the target student model after completing one round of iteration. The semantic classification module is used to acquire the data stream to be processed, input the data stream to be processed into the target student model for calculation, and obtain the semantic classification result of the data stream to be processed.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a passive online cross-domain semantic segmentation method program for rainy days stored in the memory and executable on the processor, wherein when the passive online cross-domain semantic segmentation method program for rainy days is executed by the processor, the passive online cross-domain semantic segmentation method for rainy days implements the steps of the passive online cross-domain semantic segmentation method for rainy days as described above.

[0017] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a passive online cross-domain semantic segmentation method program for rainy days, and when the passive online cross-domain semantic segmentation method program for rainy days is executed by a processor, it implements the steps of the passive online cross-domain semantic segmentation method for rainy days as described above.

[0018] In this invention, a prior model is determined and a feature prototype is obtained. The target domain data stream is input into the teacher model and the prior model for prediction, resulting in a first normalization result. A denoised pseudo-label is then obtained based on the first normalization result. The target domain data stream is input into the student model, and a segmentation loss is obtained based on the student model's prediction result and the denoised pseudo-label. A second normalization result is obtained based on the student model's prediction result and the feature prototype. A feature space structure consistency constraint is obtained based on the second and first normalization results. The model is then trained under supervision based on the segmentation loss and the feature space structure consistency constraint to obtain a target student model. The data stream to be processed is input into the target student model to obtain the semantic classification result of the data stream. The confidence-based model parameter mixing strategy proposed in this invention can effectively suppress catastrophic forgetting without using source domain data and can achieve accurate semantic category segmentation even in rainy weather. Attached Figure Description

[0019] Figure 1 This is a flowchart of a preferred embodiment of the passive online cross-domain semantic segmentation method for rainy days according to the present invention; Figure 2 This is the overall framework diagram of the passive online cross-domain semantic segmentation method for rainy days according to the present invention; Figure 3 This is a schematic diagram illustrating the applicable task scenario for the passive online cross-domain semantic segmentation method of the present invention used in rainy weather; Figure 4 This is a flowchart of the pseudo-label bidirectional denoising method based on category prototype in the passive online cross-domain semantic segmentation method for rainy days of the present invention; Figure 5 This is a schematic diagram of the confidence-based model parameter mixing strategy in the passive online cross-domain semantic segmentation method for rainy days of the present invention; Figure 6 This is a schematic diagram illustrating the principle of a preferred embodiment of the passive online cross-domain semantic segmentation method system for rainy days according to the present invention; Figure 7 This is a schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0020] This application provides a passive online cross-domain semantic segmentation method and related equipment for rainy weather. To make the purpose, technical solution, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0021] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0022] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0023] The passive online cross-domain semantic segmentation method for rainy days described in the preferred embodiment of the present invention, such as... Figure 1 and Figure 2 As shown, the passive online cross-domain semantic segmentation method for rainy days includes the following steps: Step S10: Obtain the target domain data stream for rainy days and the source domain data stream for non-rainy days. Input the source domain data stream and the target domain data stream into the source pre-trained model for calculation to obtain the first target domain class feature prototype, the confidence of each batch of images in the source domain and the confidence of each batch of images in the target domain. Obtain the prior model based on the confidence of each batch of images in the target domain.

[0024] Currently, most traditional semantic segmentation algorithms based on unsupervised domain adaptation suffer from two main problems. First, these methods typically require complete knowledge of both the source and target domains and necessitate training over multiple epochs during the domain adaptation process. Second, these methods largely assume a specific target domain and have limited long-term continuous domain adaptation capabilities. This makes them ill-suited for real-world scenarios such as rainy weather, where the complete target domain information is unknown, rainfall intensity varies continuously, data arrives in streaming form, and the source domain may be inaccessible due to privacy or confidentiality concerns. In other words, existing semantic segmentation models can perform good image recognition and classification when the input data stream is images acquired on sunny days, but they fail to accurately recognize and classify images acquired on rainy days or other adverse environmental conditions.

[0025] To adapt to real-world needs, developing an online continuous domain adaptation semantic segmentation algorithm that can achieve rapid adaptation without relying on source domain data during the domain adaptation process, and only uses the target domain data stream for one epoch of training, while possessing continuous continuous domain adaptation capabilities, is crucial for the application of semantic segmentation algorithms in rainy scenarios. Figure 3 As shown, the task scenarios applicable to the passive online continuous domain adaptive semantic segmentation algorithm of the present invention are presented.

[0026] It is understood that the target domain data stream on rainy days and the source domain data stream on non-rainy days are both data streams from the same location, only the weather conditions at the time of acquisition are different.

[0027] Specifically, the source domain data stream is input into the source pre-trained model in batches. Feature extraction is performed to obtain the extracted features. Based on the extracted features The first power prototype of the source domain class features is calculated. The quadratic prototype of source domain class features ; Among them, the first-order prototype of the source domain class features is calculated. The process is as follows: ; ; in, This represents the class prototypes of a batch of images in the source domain, i.e., the mean of the class features of the batch of images. The labels predicted by the source pre-trained model. This indicates that the source domain images in the current batch belong to class [class name missing]. The number of pixels, For batch size, The ordinal number of the batch. The height of the image. The width of the image. This is the prototype momentum coefficient. Similarly, the quadratic prototype of the source domain class features. It can also be calculated using the formula above, which will not be elaborated here.

[0028] Furthermore, the target domain data stream is input into the source pre-trained model in batches. Calculations are performed to obtain the confidence score for each batch of images in the target domain. ,in, For the batch ordinal number: ; ; in, This represents the total number of pixels in a batch of images. For soft prediction of the source pre-trained model, The length of the sliding window.

[0029] Understandably, to obtain a more robust representation, a sliding window algorithm is used here to calculate the length. The mean confidence score within the window represents the confidence score of the source pre-trained model for the current batch. Confidence level A larger value indicates that the distribution of the current batch of images is closer to the source domain, and vice versa.

[0030] Furthermore, based on the confidence level of each batch of images in the source domain... The prior model is obtained. : ; in, and To represent different confidence thresholds, The threshold for confidence changes when the domain distribution changes. Indicates the first Confidence level of the batch This is the student model for the front domain.

[0031] Based on the above formula, it can be understood that when At that time, the distribution of the target domain changed. The positive or negative sign of indicates the direction of change in the target domain. A positive value indicates that the distribution of the target domain is moving closer to the source domain, while a negative value indicates that the distribution of the target domain is moving away from the source domain. When the distribution of the target domain changes, the parameters of the current student model are assigned to . Thus, the student model of the pre-domain is obtained. Its parameters remain constant as it adapts to the current target domain. This formula indicates that when the target domain distribution is very close to the source domain distribution, or when the target domain distribution differs from the source domain distribution but is moving closer to the source domain distribution, i.e. At that time, the prior model is the source pre-trained model. However, when the target domain distribution differs significantly from the source domain distribution, or when the target domain distribution differs somewhat from the source domain distribution but is moving away from the source domain distribution, i.e. At that time, the prior model is the student model of the preceding domain. .

[0032] Furthermore, based on the first power prototype of the source domain class features and the quadratic prototype of the source domain class features The prototype of the first target domain class feature is obtained, wherein the prototype of the first target domain class feature includes the prototype of the first power of the first target domain class feature. and the quadratic prototype of the first target domain class features Since the complete information of the target domain cannot be predicted in the online domain adaptation task, this algorithm uses the first power prototype of the source domain class features calculated during the preparation process. The prototype of the quadratic features of the source domain class Initialize the class feature prototype of the target domain The prototype of the quadratic features of the target domain class .

[0033] Step S20: Input the target domain data stream into the teacher model to extract features and obtain the first feature, and input the target domain data stream into the prior model for prediction to obtain the prior prediction result. Obtain the first normalization result based on the first feature and the first target domain class feature prototype. Denoise the prior prediction result based on the first normalization result to obtain the denoised pseudo label.

[0034] It is understood that the teacher model is defined as the student model. Exponential Moving Average (MEA).

[0035] like Figure 4 As shown, the step of obtaining a first normalization result based on the first feature and the first target domain class feature prototype, and then denoising the prior prediction result based on the first normalization result to obtain a denoised pseudo-label, specifically includes: Based on the first power prototype of the first target domain class features and the quadratic prototype of the first target domain class features Calculate the variance of the target domain category prototype. : According to the variance Computational Teacher Model The first feature extracted With category prototype The first normalized result of the distance between .

[0036] Specifically, the variance of the target domain category prototype is calculated according to the following formula. : .

[0037] The first normalization result is calculated according to the following formula. : ; in, To set a minimum value to prevent variance It is 0. Among them, and The distance between them is represented by the Manhattan distance. It is an exponential function. This represents the total number of classes in the current batch. Similarly, this formula can be used to obtain the normalized variance of the distance between the features extracted by the student model and the class prototypes. (Second normalization result).

[0038] Furthermore, using the obtained Prior predictions generated by the prior model Correction is performed by reweighting each channel of the prior prediction to generate pseudo-labels with less noise. .

[0039] Specifically, using the first normalization result The prior prediction results generated by the prior model Denoising is performed to obtain denoised pseudo-labels. : ; in, This is a function that converts soft tags to one-hot encoded tags.

[0040] It is understandable that the denoised pseudo-tags obtained by this invention Compared to the pseudo-labels generated by the teacher model, There is less noise in the data, therefore, the algorithm uses the denoised pseudo-labels. Guide a batch of first power prototypes of target domain image class features The quadratic prototype of class features The calculation of the pseudo-labels generated by the teacher model, rather than directly using the pseudo-labels generated by the teacher model, can achieve better training results.

[0041] Furthermore, the step of denoising the prior prediction result based on the first normalization result to obtain a denoised pseudo-label further includes: Use the aforementioned noise-reducing pseudo-labels Calculate the prototype of the true first power class feature and the prototype of the real quadratic class features ; Based on the true first power class feature prototype and the real quadratic class feature prototype The first power prototype of the first target domain class features The prototype of the square of the features of the first target domain class The update is performed to obtain the first power prototype of the second target domain class feature. Prototype of the square of the second target domain class features : ; ; in, This is the prototype momentum coefficient.

[0042] It is understood that the present invention uses the above formula to represent the first power prototype of the first target domain class feature. The prototype of the square of the features of the first target domain class This process of updating creates a mutually reinforcing relationship between the prototype and the pseudo-label, resulting in pseudo-labels with less noise and more accurate category prototypes.

[0043] Step S30: Input the target domain data stream into the student model to obtain the student model prediction result and the second feature. Calculate the segmentation loss based on the student model prediction result and the denoised pseudo-label. Obtain the second normalization result based on the second feature and the first target domain class feature prototype. Calculate the feature space structure consistency constraint based on the second normalization result and the first normalization result.

[0044] Specifically, based on the student model prediction results and the denoised pseudo-labels Calculate the segmentation loss: ; in, For cross-entropy loss, and This is a hyperparameter.

[0045] Furthermore, due to the advantages of a compact feature space, such as good intra-class similarity, inter-class separability, and clear decision boundaries, good intra-class similarity means that similar unlabeled samples may be assigned similar labels when generating pseudo-labels, thereby improving the consistency and accuracy of pseudo-labels. Good inter-class separability helps to identify and correct potential class confusion or misclassification in pseudo-labels, thus improving the quality of pseudo-labels. Clear decision boundaries help to more accurately assign the correct category to unlabeled samples and reduce ambiguity and vagueness in pseudo-labels. Therefore, a compact feature space is conducive to the refinement of pseudo-labels, producing high-quality pseudo-labels. The average teacher model is an exponential moving average of the student model, and its parameters are smoothed to help reduce model noise and improve stability. Therefore, the feature space extracted by the average teacher model is more stable and compact than that extracted by the student model.

[0046] Inspired by this, this invention proposes a feature space structure consistency constraint, which normalizes the variance of the distance between the features extracted by the student model and the class prototype by minimizing the variance. Variance normalization of the distance between features extracted by the teacher model and the category prototype Between KL Divergence forces the feature space extracted by the student model to structurally approximate the feature space extracted by the teacher model.

[0047] Specifically, a second normalization result is obtained based on the second feature and the first target domain class feature prototype, and the second normalization result is then... Compared with the first normalized result Between KL Minimizing the divergence yields the structural consistency constraint of the feature space. : ; in, KL The divergence between the normalization results, This represents the total number of categories in the current batch. c The ordinal number of the class. For the first c The first normalization result of the class feature prototype, For the first c The second normalization result of the class feature prototype.

[0048] Step S40: Calculate the target loss function based on the segmentation loss and the feature space structure consistency constraint, train the student model by minimizing the target loss function to obtain the student model after one iteration, and update the parameters of the teacher model by exponential moving average.

[0049] Specifically, the target loss function is calculated based on the segmentation loss and the feature space structure consistency constraint. : .

[0050] Furthermore, the training of the student model is supervised by minimizing the total loss. When the target loss function reaches a preset threshold, the student model after the first iteration is output. Based on the student model after the first iteration, the parameters of the teacher model are updated by exponential moving average.

[0051] Step S50: Adjust the parameters of the student model after the first iteration based on the confidence scores of each batch of images in the target domain and the source domain, and then perform the next iteration on the adjusted student model to obtain the target student model after completing one round of iteration.

[0052] like Figure 5 As shown, in the model parameter mixing stage, the proposed confidence-based model parameter mixing strategy is used to determine whether to mix the parameters of the source pre-trained model and the parameters of the current student model.

[0053] Specifically, the target domain data stream is input into the source pre-model. Perform calculations to obtain the source pre-trained model. Confidence level of the target domain images in the current batch ; Based on the confidence level of the target domain images in the current batch Calculate the initial parameters and mixed momentum : ; Furthermore, based on the distribution relationship between the target domain and the source domain, different attenuation coefficients are used to adjust the initial parameter mixing momentum. The target mixed momentum is obtained by attenuation. ; ; in, The source domain confidence threshold. The value is the confidence score of each batch of source domain images calculated during the preparation process. The mean confidence level of the top 20 values ​​sorted from smallest to largest.

[0054] Understandable, This indicates that the current target domain can be considered as the source domain. and The coefficient of decrease of the mixed momentum. for The minimum of the mixed momentum at time t. This formula indicates that the target domain distribution and the source domain distribution have some differences, but their distributions are changing and the direction of change is towards the source domain distribution, i.e. ( Figure 4 When the initial mixed momentum is in the orange and red portions of the medium confidence curve, The value is positive if the target domain is positive and zero otherwise. During the adaptation process, different attenuation coefficients are used based on the distance between the target domain and the source domain to gradually attenuate the mixed momentum. When the target domain is the source domain, the initial mixed momentum is relatively large, and the attenuation coefficient is also relatively large. Conversely, when the target domain is some distance from the source domain, the initial mixed momentum is relatively small, and the attenuation coefficient is also relatively small.

[0055] Furthermore, based on the target mixed momentum The parameters of the student model after the first iteration are adjusted as follows: ; in, For the parameters of the adjusted student model, The parameters of the source pre-trained model, These are the parameters of the student model after the first iteration.

[0056] Understandably, based on the target mixed momentum After adjusting the student model parameters following the first iteration to obtain the adjusted student model, the second iteration of the model continues; since the first power prototype of the first target domain class features has already been obtained... The prototype of the square of the features of the first target domain class The update yielded a more accurate first-order prototype of the second target domain class features. Prototype of the square of the second target domain class features Therefore, starting from the second iteration, the first power prototype of the second target domain class feature is used. The prototype of the square of the second target domain class features Calculate the first normalization result, and update the first power prototype of the second target domain class feature after each iteration. Prototype of the square of the second target domain class features .

[0057] Step S60: Obtain the data stream to be processed, input the data stream to be processed into the target student model for calculation, and obtain the semantic classification result of the data stream to be processed.

[0058] Specifically, the target domain data stream to be processed is output to the target student model after training for calculation, which can output the semantic analysis results of the image under rainy weather conditions (e.g., accurately distinguishing vehicles, trees or pedestrians in the current scene).

[0059] As can be seen, this invention uses variance normalization of the distance between the average teacher-extracted features and the prototype to reweight the prior predictions, effectively reducing noise in the original prior predictions. Simultaneously, it imposes structural consistency constraints on the feature space extracted by the student model, forcing the student model's feature space to structurally approximate the compact feature space extracted by the teacher model, thereby further refining the pseudo-labels and obtaining high-quality pseudo-labels to supervise the training of the student model. Furthermore, compared to methods that suppress catastrophic forgetting by replaying source domain data, the confidence-based model parameter mixing strategy proposed in this invention can effectively suppress catastrophic forgetting without using source domain data. This broadens the application scenarios of this invention, making it suitable for situations where source domain data is limited and inaccessible, and also saves storage space on deployment terminals, enabling accurate semantic category segmentation even in rainy weather.

[0060] Furthermore, such as Figure 6 As shown, based on the above-mentioned passive online cross-domain semantic segmentation method for rainy days, this invention also provides a corresponding passive online cross-domain semantic segmentation method system for rainy days, wherein the passive online cross-domain semantic segmentation method system for rainy days includes: The data preparation module 51 is used to acquire the target domain data stream on rainy days and the source domain data stream on non-rainy days. The source domain data stream and the target domain data stream are respectively input into the source pre-trained model for calculation to obtain the first target domain class feature prototype, the confidence of each batch of images in the source domain and the confidence of each batch of images in the target domain. The prior model is obtained based on the confidence of each batch of images in the target domain. The pseudo-label denoising module 52 is used to input the target domain data stream into the teacher model for feature extraction to obtain the first feature, input the target domain data stream into the prior model for prediction to obtain the prior prediction result, obtain the first normalization result based on the first feature and the first target domain class feature prototype, and denoise the prior prediction result based on the first normalization result to obtain the denoised pseudo-label. The loss calculation module 53 is used to input the target domain data stream into the student model to obtain the student model prediction result and the second feature, calculate the segmentation loss based on the student model prediction result and the denoised pseudo label, obtain the second normalization result based on the second feature and the first target domain class feature prototype, and calculate the feature space structure consistency constraint based on the second normalization result and the first normalization result. The model update module 54 is used to calculate the target loss function based on the segmentation loss and the feature space structure consistency constraint, train the student model by minimizing the target loss function to obtain the student model after one iteration, and update the parameters of the teacher model by exponential moving average. The target model acquisition module 55 is used to adjust the parameters of the student model after one iteration based on the confidence scores of each batch of images in the target domain and the confidence scores of each batch of images in the source domain, and then perform the next iteration on the adjusted student model to obtain the target student model after completing one round of iteration. The semantic classification module 56 is used to acquire the data stream to be processed, input the data stream to be processed into the target student model for calculation, and obtain the semantic classification result of the data stream to be processed.

[0061] Furthermore, such as Figure 7 As shown, based on the above-mentioned passive online cross-domain semantic segmentation method and system for rainy days, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 7 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0062] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a passive online cross-domain semantic segmentation method program 40 for rainy days. This passive online cross-domain semantic segmentation method program 40 for rainy days can be executed by the processor 10, thereby implementing the passive online cross-domain semantic segmentation method for rainy days in this application.

[0063] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the passive online cross-domain semantic segmentation method for rainy days.

[0064] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0065] In one embodiment, when processor 10 executes program 40 of passive online cross-domain semantic segmentation method for rainy days in memory 20, the following steps are performed: Acquire the target domain data stream for rainy days and the source domain data stream for non-rainy days. Input the source domain data stream and the target domain data stream into the source pre-trained model for calculation to obtain the first target domain class feature prototype, the confidence of each batch of images in the source domain and the confidence of each batch of images in the target domain. Obtain the prior model based on the confidence of each batch of images in the target domain. The target domain data stream is input into the teacher model for feature extraction to obtain the first feature, and the target domain data stream is input into the prior model for prediction to obtain the prior prediction result. The first normalization result is obtained based on the first feature and the first target domain class feature prototype. The prior prediction result is denoised based on the first normalization result to obtain the denoised pseudo label. The target domain data stream is input into the student model to obtain the student model prediction result and the second feature. The segmentation loss is calculated based on the student model prediction result and the denoised pseudo-label. The second normalization result is obtained based on the second feature and the first target domain class feature prototype. The feature space structure consistency constraint is calculated based on the second normalization result and the first normalization result. The target loss function is calculated based on the segmentation loss and the feature space structure consistency constraint. The student model is trained by minimizing the target loss function to obtain the student model after one iteration. The parameters of the teacher model are updated by exponential moving average. The parameters of the student model after the first iteration are adjusted based on the confidence scores of each batch of images in the target domain and the confidence scores of each batch of images in the source domain. The adjusted student model is then used for the next iteration. After completing one round of iterations, the target student model is obtained. Obtain the data stream to be processed, input the data stream to be processed into the target student model for calculation, and obtain the semantic classification result of the data stream to be processed.

[0066] Specifically, the step of inputting the source domain data stream into the source pre-trained model for calculation, determining the prior model based on the calculation results, and obtaining the first target domain class feature prototype includes: The source domain data stream is input into the source pre-trained model in batches. Feature extraction is performed, and the first power prototype of the source domain class features is calculated based on the extracted features. The quadratic prototype of source domain class features ; The target domain data stream is input into the source pre-trained model in batches. Calculations are performed to obtain the confidence score for each batch of images in the target domain. ,in, The ordinal number of the batch; Based on the confidence level of each batch of images in the target domain The prior model is obtained. : ; in, and To represent different confidence thresholds, The threshold for confidence changes when the domain distribution changes. Indicates the first Confidence level of the batch For the student model in the front domain; Based on the first power prototype of the source domain class characteristics and the quadratic prototype of the source domain class features The prototype of the first target domain class feature is obtained, wherein the prototype of the first target domain class feature includes the prototype of the first power of the first target domain class feature. and the quadratic prototype of the first target domain class features .

[0067] The teacher model is defined as the student model. Exponential moving average; The step of obtaining the first normalization result based on the first feature and the first target domain class feature prototype specifically includes: Based on the first power prototype of the first target domain class features and the quadratic prototype of the first target domain class features Calculate the variance of the target domain category prototype. : According to the variance Computational Teacher Model The first feature extracted With category prototype The first normalized result of the distance between .

[0068] Specifically, the step of denoising the prior prediction result based on the first normalization result to obtain denoised pseudo-labels includes: Use the first normalization result The prior prediction results generated by the prior model Denoising is performed to obtain denoised pseudo-labels. : ; in, This is a function that converts soft tags to one-hot encoded tags.

[0069] The step of denoising the prior prediction result based on the first normalization result to obtain a denoised pseudo-label further includes: Use the aforementioned noise-reducing pseudo-labels Calculate the prototype of the true first power class feature and the prototype of the real quadratic class features ; Based on the true first power class feature prototype and the real quadratic class feature prototype The first power prototype of the first target domain class features The prototype of the square of the features of the first target domain class The update is performed to obtain the first power prototype of the second target domain class feature. Prototype of the square of the second target domain class features : ; ; in, This is the prototype momentum coefficient.

[0070] Specifically, the step of calculating the segmentation loss based on the student model prediction result and the denoised pseudo-label, obtaining the second normalization result based on the second feature and the first target domain class feature prototype, and calculating the feature space structure consistency constraint based on the second normalization result and the first normalization result, includes: Based on the student model prediction results and the denoised pseudo-labels Calculate the segmentation loss: ; in, For cross-entropy loss, and For hyperparameters; A second normalization result is obtained based on the second feature and the first target domain class feature prototype. The second normalization result... Compared with the first normalized result Between KL Minimizing the divergence yields the structural consistency constraint of the feature space. : ; in, KL The divergence between the normalization results, This represents the total number of categories in the current batch. c The ordinal number of the class. For the first c The first normalization result of the class feature prototype, For the first c The second normalization result of the class feature prototype.

[0071] The passive online cross-domain semantic segmentation method for rainy days further includes: Starting from the second iteration, use the first power prototype of the second target domain class feature. The prototype of the square of the second target domain class features Calculate the first normalization result, and update the first power prototype of the second target domain class feature after each iteration. Prototype of the square of the second target domain class features .

[0072] Specifically, adjusting the student model parameters after the first iteration based on the confidence scores of each batch of images in the target domain and the source domain includes: The initial parameter mixed momentum is calculated based on the confidence scores of each batch of images in the target domain and each batch of images in the source domain. Based on the distribution relationship between the target domain and the source domain, different attenuation coefficients are used to adjust the mixed momentum of the initial parameters. The target mixed momentum is obtained by attenuation. ; Based on the target mixed momentum The parameters of the student model after the first iteration are adjusted as follows: ; in, For the parameters of the adjusted student model, The parameters of the source pre-trained model, These are the parameters of the student model after the first iteration.

[0073] In summary, this invention provides a passive online cross-domain semantic segmentation method and related equipment for rainy days. The method includes: determining a prior model and obtaining a feature prototype; inputting the target domain data stream into a teacher model and a prior model for prediction, obtaining a first normalization result, and obtaining a denoised pseudo-label based on the first normalization result; inputting the target domain data stream into a student model, obtaining a segmentation loss based on the student model's prediction result and the denoised pseudo-label, obtaining a second normalization result based on the student model's prediction result and the feature prototype, and obtaining a feature space structure consistency constraint based on the second normalization result and the first normalization result; supervising the model training based on the segmentation loss and the feature space structure consistency constraint to obtain a target student model; and inputting the data stream to be processed into the target student model to obtain a semantic classification result of the data stream to be processed. The model provided by this invention can achieve accurate semantic category segmentation in rainy days.

[0074] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0075] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A passive online cross-domain semantic segmentation method for rainy days, characterized in that, The passive online cross-domain semantic segmentation method for rainy days includes: Acquire the target domain data stream for rainy days and the source domain data stream for non-rainy days. Input the source domain data stream and the target domain data stream into the source pre-trained model for calculation to obtain the first target domain class feature prototype, the confidence of each batch of images in the source domain and the confidence of each batch of images in the target domain. Obtain the prior model based on the confidence of each batch of images in the target domain. The target domain data stream is input into the teacher model for feature extraction to obtain the first feature, and the target domain data stream is input into the prior model for prediction to obtain the prior prediction result. The first normalization result is obtained based on the first feature and the first target domain class feature prototype. The prior prediction result is denoised based on the first normalization result to obtain the denoised pseudo label. The target domain data stream is input into the student model to obtain the student model prediction result and the second feature. The segmentation loss is calculated based on the student model prediction result and the denoised pseudo-label. The second normalization result is obtained based on the second feature and the first target domain class feature prototype. The feature space structure consistency constraint is calculated based on the second normalization result and the first normalization result. The target loss function is calculated based on the segmentation loss and the feature space structure consistency constraint. The student model is trained by minimizing the target loss function to obtain the student model after the first iteration. The parameters of the teacher model are updated by exponential moving average. The parameters of the student model after the first iteration are adjusted based on the confidence scores of each batch of images in the target domain and the confidence scores of each batch of images in the source domain. The adjusted student model is then used for the next iteration. After one round of iteration, the target student model is obtained. Obtain the data stream to be processed, input the data stream to be processed into the target student model for calculation, and obtain the semantic classification result of the data stream to be processed.

2. The passive online cross-domain semantic segmentation method for rainy days according to claim 1, characterized in that, The process of inputting the source domain data stream and the target domain data stream into the source pre-trained model for calculation to obtain the first target domain class feature prototype, the confidence score of each batch of images in the source domain, and the confidence score of each batch of images in the target domain, and obtaining the prior model based on the confidence score of each batch of images in the target domain, specifically includes: The source domain data stream is input into the source pre-trained model in batches. Feature extraction is performed, and the first power prototype of the source domain class features is calculated based on the extracted features. The quadratic prototype of source domain class features ; The target domain data stream is input into the source pre-trained model in batches. Calculations are performed to obtain the confidence score for each batch of images in the target domain. ,in, The ordinal number of the batch; Based on the confidence level of each batch of images in the target domain The prior model is obtained. : ； in, and To represent different confidence thresholds, The threshold for confidence changes when the domain distribution changes. Indicates the first Confidence level of the batch For the student model in the front domain; Based on the first power prototype of the source domain class characteristics and the quadratic prototype of the source domain class features The prototype of the first target domain class feature is obtained, wherein the prototype of the first target domain class feature includes the prototype of the first power of the first target domain class feature. and the quadratic prototype of the first target domain class features .

3. The passive online cross-domain semantic segmentation method for rainy days according to claim 2, characterized in that, The teacher model is defined as the student model. Exponential moving average; The step of obtaining the first normalization result based on the first feature and the first target domain class feature prototype specifically includes: Based on the first power prototype of the first target domain class features and the quadratic prototype of the first target domain class features Calculate the variance of the target domain category prototype. : According to the variance Computational Teacher Model The first feature extracted With category prototype The first normalized result of the distance between .

4. The passive online cross-domain semantic segmentation method for rainy days according to claim 3, characterized in that, The step of denoising the prior prediction result based on the first normalization result to obtain denoised pseudo-labels specifically includes: Use the first normalization result The prior prediction results generated by the prior model Denoising is performed to obtain denoised pseudo-labels. : ； in, This is a function that converts soft tags to one-hot encoded tags.

5. The passive online cross-domain semantic segmentation method for rainy days according to claim 4, characterized in that, The step of denoising the prior prediction result based on the first normalization result to obtain a denoised pseudo-label further includes: Use the aforementioned noise-reducing pseudo-labels Calculate the prototype of the true first power class feature and the prototype of the real quadratic class features ; Based on the true first power class feature prototype and the real quadratic class feature prototype The first power prototype of the first target domain class features The prototype of the square of the features of the first target domain class The update is performed to obtain the first power prototype of the second target domain class feature. Prototype of the square of the second target domain class features : ； ； in, This is the prototype momentum coefficient.

6. The passive online cross-domain semantic segmentation method for rainy days according to claim 4, characterized in that, The step involves calculating a segmentation loss based on the student model prediction results and the denoised pseudo-labels, obtaining a second normalization result based on the second feature and the first target domain class feature prototype, and calculating a feature space structure consistency constraint based on the second normalization result and the first normalization result. Specifically, this includes: Based on the student model prediction results and the denoised pseudo-labels Calculate the segmentation loss: ； in, For cross-entropy loss, and For hyperparameters; A second normalization result is obtained based on the second feature and the first target domain class feature prototype. The second normalization result... Compared with the first normalized result Between KL Minimizing the divergence yields the structural consistency constraint of the feature space. : ； in, KL The divergence between the normalization results, This represents the total number of categories in the current batch. c The ordinal number of the class. For the first c The first normalization result of the class feature prototype, For the first c The second normalization result of the class feature prototype.

7. The passive online cross-domain semantic segmentation method for rainy days according to claim 5, characterized in that, The passive online cross-domain semantic segmentation method for rainy days also includes: Starting from the second iteration, use the first power prototype of the second target domain class feature. The prototype of the square of the second target domain class features Calculate the first normalization result, and update the first power prototype of the second target domain class feature after each iteration. Prototype of the square of the second target domain class features .

8. The passive online cross-domain semantic segmentation method for rainy days according to claim 1, characterized in that, The step of adjusting the student model parameters after the first iteration based on the confidence scores of each batch of images in the target domain and the source domain specifically includes: The initial parameter mixed momentum is calculated based on the confidence scores of each batch of images in the target domain and each batch of images in the source domain. Based on the distribution relationship between the target domain and the source domain, different attenuation coefficients are used to adjust the mixed momentum of the initial parameters. The target mixed momentum is obtained by attenuation. ; Based on the target mixed momentum The parameters of the student model after the first iteration are adjusted as follows: ； in, For the parameters of the adjusted student model, The parameters of the source pre-trained model, These are the parameters of the student model after the first iteration.

9. A passive online cross-domain semantic segmentation method system for rainy days, characterized in that, The passive online cross-domain semantic segmentation method system for rainy days includes: The data preparation module is used to acquire the target domain data stream during rainy days and the source domain data stream during non-rainy days. The source domain data stream and the target domain data stream are respectively input into the source pre-trained model for calculation to obtain the first target domain class feature prototype, the confidence of each batch of images in the source domain and the confidence of each batch of images in the target domain. The prior model is obtained based on the confidence of each batch of images in the target domain. The pseudo-label denoising module is used to input the target domain data stream into the teacher model for feature extraction to obtain the first feature, and input the target domain data stream into the prior model for prediction to obtain the prior prediction result. Based on the first feature and the first target domain class feature prototype, a first normalization result is obtained. Based on the first normalization result, the prior prediction result is denoised to obtain a denoised pseudo-label. The loss calculation module is used to input the target domain data stream into the student model to obtain the student model prediction result and the second feature, calculate the segmentation loss based on the student model prediction result and the denoised pseudo-label, obtain the second normalization result based on the second feature and the first target domain class feature prototype, and calculate the feature space structure consistency constraint based on the second normalization result and the first normalization result. The model update module is used to calculate the target loss function based on the segmentation loss and the feature space structure consistency constraint, train the student model by minimizing the target loss function to obtain the student model after the first iteration, and update the parameters of the teacher model by exponential moving average. The target model acquisition module is used to adjust the parameters of the student model after the first iteration based on the confidence scores of each batch of images in the target domain and the confidence scores of each batch of images in the source domain, and then perform the next iteration on the adjusted student model to obtain the target student model after completing one round of iteration. The semantic classification module is used to acquire the data stream to be processed, input the data stream to be processed into the target student model for calculation, and obtain the semantic classification result of the data stream to be processed.

10. A terminal, characterized in that, The terminal includes: a memory, a processor, and a passive online cross-domain semantic segmentation method program for rainy days stored in the memory and executable on the processor. When the passive online cross-domain semantic segmentation method program for rainy days is executed by the processor, it implements the steps of the passive online cross-domain semantic segmentation method for rainy days as described in any one of claims 1-8.

Images